PaperlayerView exact revision on arXiv

Source-linked AI summary

ReDence: A Hybrid Agentic and Decision-Model Framework for Rapid, Evidence-Grounded Detection of Modern Social Engineering Scams

Jaishnoor Kaur, Josef Horáček, Ian Waskom, Sayak Saha Roy

arXiv:2610.30963v1cs.CRcs.LG

TL;DR

Existing website-level scam defenses offer limited support for investigations spanning workflows, multiple pages, and external claim verification. ReDence combines an agentic Investigator with a specialized Decider and produces evidence-grounded reports. It reports 96% detection accuracy across twelve scam categories while outperforming evaluated detectors and running at least four times faster than comparable agentic configurations.

  • Problem

    Many SE scam investigations require examining website workflows and evidence across pages and external sources, while existing defenses provide limited investigative support.

    Source: §1 · p. 1

  • Method

    ReDence combines a fine-tuned Investigator for planning and evidence interpretation with a specialized Decider for binary and multiple-choice decisions, and records investigations in evidence-grounded reports.

    Source: §1 · p. 2

  • Results

    96% detection accuracy across twelve SE scam categories, outperforming evaluated automated detectors and completing investigations at least four times faster than comparable agentic configurations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ReDence’s structured reports record actions, artifacts, supporting and contradictory evidence, external verification, and reasoning so analysts can inspect how assessments were reached.

    Source: §1 · p. 2

    The report-quality evaluation described uses one coder; the authors are repeating the analysis with two independent coders to measure intercoder agreement.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

Detecting online social engineering (SE) scams often requires examining website workflows, verifying claims against external sources, and connecting evidence across multiple pages. Existing detectors provide limited support for these investigations, leaving analysts to manually gather and validate evidence, which inevitably delays detection and subsequent takedown of these attacks. To address this gap, we present ReDence, an open-source hybrid framework for adaptive, evidence-grounded detection across diverse SE scam categories. ReDence combines two locally hosted models with complementary responsibilities: a fine-tuned multimodal agentic Investigator (Qwen3.5-4B) that performs investigative planning, evidence interpretation, and reporting, and a specialized Decider (Tev1) that handles binary and multiple-choice tasks, including browser-action selection, evidence assessment, and decisions about whether to continue an investigation. By delegating discrete decisions to a specialized model, ReDence reduces reliance on lengthy language-model responses, while preserving the reasoning capabilities required for complex investigations. Our evaluation demonstrates that ReDence achieves 96\% accuracy in detecting scams across twelve SE scam categories, substantially outperforming existing detectors while being at least four times faster than comparable agentic models without compromising investigative depth. ReDence also produces structured, evidence-grounded reports that document investigative actions, supporting and contradictory evidence, and collected artifacts, which allow analysts to independently audit detection decisions and utilize the collected evidence to support subsequent mitigation and takedown efforts.

1. Introduction

Automated website defenses have largely centered on credential phishing, while diverse social-engineering scams demand investigations across workflows, pages, artifacts, and external sources. ReDence combines agentic investigation with specialized decision-making and produces auditable reports; its evaluation reports 96% accuracy across twelve categories and faster investigations than comparable agentic configurations.

1. Introduction

  • Credential phishing is only one part of the broader scam landscape, and detecting other scams may require multi-page navigation, workflow interaction, artifact examination, and independent claim verification.The introduction describes contradictory claims that could be uncovered by cross-referencing independent sources, including during a zero-day period.Source: §1 · p. 1
  • Existing detectors often provide only labels or confidence scores, leaving analysts to reconstruct workflows and evidence before reporting or takedown procedures.The paper notes this work is time-consuming, difficult to scale, and dependent on familiarity with scam schemes or regional contexts.Source: §1 · p. 1
  • Agentic LLMs can support in-depth investigations, but tool use alone does not ensure effective security investigation, and repeated generative decisions add latency and computational overhead.Domain-specific guidance is needed to apply relevant knowledge, recognize deception, verify claims, and distinguish suspicious observations from substantiated findings.Source: §1 · pp. 1–2
  • ReDence pairs a fine-tuned multimodal Investigator for planning, reasoning, and evidence interpretation with a specialized Decision model for binary and multiple-choice decisions.Prompts and tools support systematic exploration of websites and associated internal and external artifacts, with strategies adapting to emerging findings.Source: §1 · p. 2
  • Its reports record investigative actions, artifacts, supporting and contradictory evidence, external verification, and the reasoning behind assessments, enabling analyst inspection of conclusions and evidence.The framework aims to combine automated scalability with adaptive investigation and evidentiary records traditionally provided through human analysis.Source: §1 · p. 2
  • 96% detection accuracy across twelve SE scam categories, outperforming evaluated automated detectors and completing investigations at least four times faster than comparable agentic configurations.The framework reports no compromise in investigative depth.Source: §1 · p. 2

2. Background

Prior scam detectors vary in category coverage and investigative depth, while decision models offer compact choices for recurring bounded tasks. ReDence uses a specialized Decider for discrete choices while the Investigator retains open-ended planning, interpretation, and reporting.

2. Background

  • Existing automated scam detectors differ in coverage and depth: BeyondPhish classifies from predefined features, ScamMagnifier investigates shopping workflows, and LOKI [15] discovers candidate sites through search queries.Their respective scopes are feature-based classification, fraudulent shopping investigations, and candidate website discovery.Source: §2 · p. 2
  • ScamFerret, the closest prior work, analyzes textual artifacts with predefined tools but cannot interact directly with website workflows or associated artifacts, and does not consistently adapt to category-specific deception indicators.The paper compares ReDence with ScamFerret in Section 7.Source: §2 · p. 2
  • Decision models score bounded answer alternatives from context and a question, returning a compact choice instead of extended generated reasoning.The background identifies action selection, request routing, and content assessment as recurring uses of this approach.Source: §2 · p. 2
  • ReDence delegates discrete decisions to the Decider while the Investigator develops plans, interprets relationships among findings, and generates evidence-grounded reports.The division is intended to combine rapid decision-making with sustained reasoning and adaptability for evaluating scams.Source: §2 · p. 3

3. Research overview

ReDence uses two compact models and transfers investigative capabilities from a larger teacher through demonstrations and fine-tuning. Its study trains on a seven-category ground-truth set and evaluates on a separate set that includes five held-out categories.

3. Research overview

  • ReDence pairs 4-billion-parameter Qwen3.5-based Investigator and Tev1-based Decider models, assigning open-ended reasoning and reporting to the former and binary or multiple-choice decisions to the latter.The overview notes that smaller models can face challenges with complex instructions, multi-step planning, tool use, and structured outputs.Source: §3 · p. 3
  • A Qwen3.5-27B Mentor generates investigative demonstrations and reports, which are used to fine-tune both smaller models for specialized reasoning and tool-use capabilities.The overview describes this as transferring capabilities while preserving the smaller models’ computational efficiency.Source: §3 · p. 3
  • The study builds ground-truth data from seven scam categories plus benign websites, then tests on an independent set that adds five categories to assess generalization.The ground-truth websites support Mentor-generated reports for training the Investigator and Decider.Source: §3 · p. 3
  • The agentic setup supplies dedicated investigative prompts and tools, then evaluates Mentor’s detection, report quality, and evidence-gathering in a case study.The overview also says this stage discusses investigation limitations motivating the hybrid architecture.Source: §3 · p. 3
  • The architecture section defines Investigator–Decider responsibilities and communication, then constructs complementary training datasets from Mentor trajectories and reports.Investigator learns investigation, reporting, and delegation; Decider learns contextual binary and multiple-choice decisions.Source: §3 · p. 3
  • The final comparison examines detection, investigation time, memory, and investigation and report quality against Mentor and existing detectors.A comparative case study examines how exploration, interaction, and external verification affect collected evidence and assessments.Source: §3 · p. 3

4. Building the SE scam corpus

Because URL-only, screenshot, and pre-extracted-feature datasets do not preserve interactive websites, the authors construct and preserve their own scam and benign website corpus. The separate test set includes held-out scam categories, and replayable snapshots support evaluation when sites are unavailable.

4. Building the SE scam corpus

  • The ground-truth corpus contains 728 unique domains from 2,175 BBB reports across seven scam categories, with websites validated by two independent coders.The collection covers online purchase, counterfeit product, advance-fee loan, investment, debt collection, employment, and sweepstakes/lottery scams.Source: §4 · p. 3
  • The independent test collection includes 1,551 websites from 2,726 BBB reports: 1,243 in seven scam categories and 308 across five held-out categories.The held-out categories are phishing, travel/vacation/timeshare, government agency impersonation, home improvement, and fake invoices/supplier bills.Source: §4 · p. 4
  • The authors add 1,200 manually validated benign Curlie websites to the ground-truth collection, spanning services corresponding to the scam categories.The listed service groups include retail, branded-product storefronts, lending, investment, employment, and contests or prize promotions.Source: §4 · p. 4
  • A separate test set avoids using a train/test split on a relatively small, class-imbalanced ground-truth set and permits evaluation on independent websites.The authors state that this allows ReDence to learn from the full ground-truth report.Source: §4 · p. 4
  • Browsertrix snapshots preserve sites that may become unavailable and support reproducibility, capturing the entry page and pages within two same-host link traversals, with a 100-page crawl cap.The archived sites had median Memento Damage 0.08 (IQR 0.03-0.15); 373, approximately 14%, of ground-truth and test websites were unavailable during evaluation.Source: §4 · p. 4

5. The Mentor Model

Mentor combines multimodal website investigation with task-specific guidance, interactive tools, external search, and evidence-grounded reporting. Its evaluation examines scam detection, an adaptive PublicTrade investigation, and report evidence coverage and auditability.

5.1. Investigative Prompt

  • Mentor’s prompt guides systematic website exploration, purpose-driven planning, adaptive investigation, and evidence-grounded reporting.The agent revises plans as observations emerge, prioritizes actions that resolve meaningful uncertainty, and records actions, evidence updates, and plan updates.Source: §5.1 · pp. 5–6

5.2. Investigation Tools

  • Fetch Page gathers HTML, text, DOM, and screenshots; Interact Page supports stateful workflows, while Web Search helps verify claims against external sources.The tools connect observations to actions and allow the agent to inspect discovered source pages for corroborating or contradictory evidence.Source: §5.2 · pp. 6–7

5.3.1. Detection.

  • Mentor identifies 589 of 728 malicious websites as malicious (80.9%), while 42 are classified as benign and 97 as inconclusive.Its median investigation takes 199 seconds and 17 tool calls per website; 68% of aggregate runtime is spent on decision-making tasks.Source: §5.3.1 · p. 7
  • Repeated unresolved deliberation occurred in 72 of 97 inconclusive investigations and 26 of 42 benign classifications; duplicate tool calls appeared in 195 investigations (26.8%).These findings motivate adding a decision model to reduce decision-making overhead and repeated calls that do not advance an investigation.Source: §5.3.1 · p. 7

5.3.2. Case Study.

  • In the PublicTrade case, Mentor finds that the claimed registration number belongs to ALLBIT LTD, whose listed retail-sales activity does not substantiate the advertised financial services.Mentor also finds no established association between PublicTrade and ALLBIT LTD, and the same number appears on two other trading sites flagged by scam trackers.Source: §5.3.2 · p. 8
  • The PublicTrade investigation combines unresolved identity claims, a reused registration number, unavailable legal pages, payment methods, and absent identity verification into a high-confidence investment or cryptocurrency scam assessment.Attempts to retrieve legal, about, terms, and privacy pages return 404 errors, while searches do not establish an association between advertised contacts and PublicTrade.Source: §5.3.2 · pp. 7–8

5.4. Evaluating Reports

  • The report evaluation compares Mentor’s findings with a cybersecurity professional’s records from 70 websites, using a balanced sample of 35 malicious and 35 benign sites.The coder investigates each site for 90 seconds before reviewing Mentor’s report; additional Mentor findings are accepted only after verification, and repeated findings are counted once.Source: §5.4 · p. 9
  • 0.87 mean evidence coverage and 0.79 auditable evidence coverage; Mentor reports add 364 findings absent from initial coder records, 327 treated as auditable.For the 52 correctly classified websites, reports average 6.8 distinct relevant findings versus 2.4 in the coder’s initial records.Source: §5.4 · p. 9

6. ReDence

ReDence divides investigation between an Investigator for open-ended reasoning and a Decider for bounded choices, trained on Mentor-generated investigative trajectories. A Controller mediates their decisions and tool use, while recording observations that feed the final evidence-grounded assessment.

6. ReDence

  • Investigator handles open-ended planning, interpretation, and reporting, while Decider resolves binary and multiple-choice decisions such as browser actions and evidence judgments.The Investigator formulates requests and uses Decider’s selections to continue its investigation.Source: §6 · pp. 9–10

6.1. Constructing the Training Datasets

  • Training datasets derive from Mentor’s investigations of malicious and benign websites, retaining sequential actions and observations and adapting trajectories for Investigator delegation.Investigator receives most of the trajectory; Decider examples pair investigative context and a question with labeled alternatives and Mentor’s recorded choice or judgment.Source: §6.1 · p. 10
  • Decider examples cover source retrieval, website interactions, bounded evidence questions, investigation continuation or revision, and final verdict and category selection.Examples are retained when context, choices, and the correct response can be reliably reconstructed from Mentor’s trajectory; insufficient-evidence and continue-investigating options may also be included.Source: §6.1 · p. 10
  • Both models are fine-tuned independently with QLoRA and frozen 4-bit backbones, using different sequence lengths, LoRA settings, epochs, and learning rates.Investigator uses a 16,384-token maximum sequence, rank 16, α = 32, three epochs, and 1 × 10−4 learning rate; Decider uses 2,048 tokens, rank 8, α = 16, one epoch, and 5 × 10−5.Source: §6.1 · p. 10

6.2. Framework Coordination with Controller

  • Controller validates Investigator’s decision requests, passes context and alternatives to Decider, and returns its choice; when needed, it executes the associated tool operation and returns the result.For example, Decider can select a menu entry, after which Controller performs the interaction and returns the updated page content to Investigator.Source: §6.2 · pp. 10–11
  • Controller records decision context, model judgments, tool operations, and observations or errors; Investigator uses these records to maintain an evidence ledger and delegates the final verdict and category selection.Controller also enforces a 120-second-per-website execution budget and checks report structure and evidence references.Source: §6.2 · p. 11

7. Evaluating ReDence

The evaluation compares ReDence with automated detectors and agentic baselines on scam detection, website investigations, and reporting. It also examines how interactive investigation changes the evidence gathered in a case study.

7. Evaluating ReDence

  • ReDence combines systematic website exploration, purpose-driven planning, adaptive investigation, and evidence-grounded reporting, with the Decider selecting actions and resolving discrete judgments.The evaluation used a separate test set of 1,551 scam and 1,500 benign websites.Source: §7 · p. 12

7.1. Detection Effectiveness

  • ReDence achieved F1 0.97 versus Mentor’s 0.82 and completed investigations for all test websites, while Mentor failed to complete 208.ReDence’s median investigation time was 43.8 seconds versus 194 seconds for Mentor, with nine versus 15 median tool calls.Source: §7.1 · p. 12, Table 4 · p. 13
  • On held-out scam categories, ReDence achieved F1 0.96 versus Mentor’s 0.82 and maintained recall of 0.98.ReDence’s precision declined only marginally on held-out categories.Source: §7.1 · p. 12
  • ScamFerret’s GPT-4.1 and Llama configurations scored F1 0.82 and 0.71; URLNet scored 0.25, while BeyondPhish scored 0.78 with recall 0.70.ScamFerret’s configurations failed to complete 115 and 239 investigations, respectively.Source: §7.1 · pp. 12–13

7.2. Case Study: wego-france.com

  • In the case study, ReDence tested nonfunctional comment controls and a resetting countdown, then uncovered a survey-to-prize workflow that requested personal and payment information.It classified the site as a sweepstakes scam; ScamFerret did not interact with the workflow and returned “Not malicious.”Source: §7.2 · pp. 14–15
  • ReDence’s report linked apparent delivery endorsements to nonclickable profiles and unresponsive controls, whereas ScamFerret did not document those interaction checks.ReDence records the observations and explains how they jointly support its interpretation.Source: §7.2 · p. 15

7.3. Reporting

  • Across 120 sampled websites, ReDence’s median EC and AEC scores were 0.87 and 0.81, compared with Mentor’s 0.84 and 0.77 and ScamFerret’s 0.44 and 0.30.For 37 correctly identified malicious sites, the coder recorded 2.8 relevant findings per site on average, versus 7.0 for ReDence, 6.5 for Mentor, and 3.4 for ScamFerret.Source: §7.3 · p. 16

8. Conclusion

ReDence’s hybrid architecture accelerates decision-making while outperforming larger and existing detectors. Its reports add evidence and context that support more informed, verifiable assessments.

8. Conclusion

  • The specialized Decider accelerates decisions and substantially reduces investigation time while ReDence outperforms its larger Mentor model and state-of-the-art scam detectors.The Investigator’s evidence-grounded reports provide findings and context analysts may have overlooked during manual investigations.Source: §8 · p. 16
  • ReDence’s reports provide analysts with additional findings and contextual information that may have been overlooked during manual investigations.These reports support more informed and verifiable assessments of website legitimacy.Source: §8 · p. 16
  • The paper concludes that ReDence’s hybrid architecture combines faster decision-making with stronger detection performance.The conclusion compares ReDence with both its larger Mentor model and state-of-the-art scam detectors.Source: §8 · p. 16

Related papers

Loading 2610.30963v1…