PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey of Data Mining-based Fraud Detection Research

Clifton Phua, Vincent Lee, Kate Smith, Ross Gayler

arXiv:1009.6119v1cs.AIcs.CE

TL;DR

Fraud detection research lacks accessible real-world datasets and faces challenges across diverse, evolving data. This survey synthesizes published data-mining methods and techniques, concluding that related fields offer promising approaches for future fraud detection research.

  • Problem

    Fraud detection research lacks publicly available datasets, while companies rarely share real data because of legal and competitive constraints.

  • Method

    The paper categorises, compares, and summarises published data-mining-based fraud detection methods and techniques across large datasets and streams.

  • Results

    The survey identifies limitations in existing fraud-detection methods and highlights unsupervised, monitoring, text-mining, semi-supervised, and game-theoretic approaches from related fields.

  • Takeaways & Limitations

    Future fraud-detection research can benefit from approaches developed in counterterrorism, law enforcement, intrusion detection, and spam detection.

  • Takeaways & Limitations

    The paper notes excessive emphasis on complex supervised algorithms despite the potential of simpler methods and the need for unsupervised approaches when labels are unavailable.

Abstract

from arXiv · show

This survey paper categorises, compares, and summarises from almost all published technical and review articles in automated fraud detection within the last 10 years. It defines the professional fraudster, formalises the main types and subtypes of known fraud, and presents the nature of data evidence collected within affected industries. Within the business context of mining the data to achieve higher cost savings, this research presents methods and techniques together with their problems. Compared to all related reviews on fraud detection, this survey covers much more technical articles and is the only one, to the best of our knowledge, which proposes alternative data and solutions from related domains.

1. INTRODUCTION & MOTIVATION

Fraud detection uses mathematical algorithms to extract possible fraud evidence from available data because transaction legitimacy and intent cannot be known with absolute certainty. This survey addresses persistent limitations in the field by categorising research, comparing methods, and identifying alternative data and techniques from related adversarial domains.

  • Motivation: Fraud detection automates and reduces manual screening within broader fraud control, while fraud becomes business-critical when prevalent and prevention procedures are not fail-safe.The paper defines fraud as abusing a profit organisation’s system without necessarily causing direct legal consequences.
  • Motivation: Because legitimacy and intention cannot be determined absolutely, cost-effective detection extracts possible fraud evidence from available data using mathematical algorithms.This frames automated detection as an evidence-discovery problem rather than a certainty-based decision process.
  • Research gaps: To address scarce public real data and insufficiently researched published methods, the paper consolidates related literature, discusses innovative techniques, and identifies alternative data sources.These two criticisms are presented as central challenges for data mining-based fraud detection research.
  • Survey objectives: The survey categorises, compares, and summarises data mining-based fraud detection methods and techniques while defining challenges across different types of large data sets and streams.It covers methods and techniques reported in published academic and industrial research.
  • Related domains: The survey highlights transferable directions from epidemic detection, insider trading, intrusion detection, money laundering, spam detection, and terrorist detection to avoid repeated mistakes and reinvention.The paper treats knowledge and experience from these adversarial data mining domains as interchangeable for fraud detection.

2. BACKGROUND

The background distinguishes internal and external fraudsters, emphasizing professional offenders’ adaptive behavior and the use of data mining to prioritize investigation. It also surveys fraud-detection domains, with credit transactional fraud receiving the most research attention.

  • 2.1 Fraudsters: Fraudsters may be internal management or non-management employees, or external customers and suppliers.External fraudsters include average, criminal, and organised crime offenders.
  • 2.1 Fraudsters: Professional fraudsters repeatedly disguise identities or evolve their modus operandi to approximate legal activity and counter detection systems.Detection therefore needs to account for strategic moves and countermoves between algorithms and offenders.
  • 2.1 Fraudsters: Data mining outputs such as suspicion scores, rules, and visual anomalies identify the riskiest external parties for investigation when manual checking is cost-prohibitive.This approach addresses companies interacting with up to millions of external parties.
  • 2.2 Affected Industries: The survey covers internal, insurance, credit card, telecommunications, e-business, e-commerce, and network-related fraud-detection applications.Internet e-business and e-commerce create a challenging data-mining task because they blur fraud detection and network intrusion detection boundaries.
  • 2.2 Affected Industries: Credit transactional fraud detection has received the most attention among the surveyed research domains.The category is used broadly here to include bankruptcy and bad-debts prediction, while employee/retail, national crop insurance, and credit application each have one publication.

3. DATA AND MEASUREMENTS

Fraud-detection studies use heterogeneous, often highly skewed structured data whose availability and realism are major constraints. Because error costs vary, research increasingly emphasizes cost-sensitive and score-based performance measures over thresholded accuracy.

  • Structured data: 16 data sets have less than 10 attributes, 18 have 10 to 49, 5 have 50 to 99, and 1 has more than 100 attributes.Attributes may be binary, numerical, categorical, or mixed.
  • Structured data: Credit datasets typically exceed 1 million transactions, with the largest exceeding 12 million per year, while management datasets contain fewer than 500 examples.Insurance datasets contain hundreds of examples, with the largest reaching 40000 examples; employee/retail data exceed 5 million transactions.
  • Structured data: More than 80 percent (16 papers) of 19 papers has skewed data with less than 30% fraud, including rates as low as 0.1 percent in credit data and 0.5 percent in home insurance data.The average proportion of test examples to total examples is around 50%.
  • Structured data: Most internal and insurance fraud data are static and lack temporal information, while some collected attributes are ineffective for fraud detection.The insurance pathology-provider data described by Yamanishi et al. lacks temporal information, and attributes such as the proportion of tests performed are not effective.
  • Data availability: Public fraud datasets are largely unavailable, real company data are difficult to obtain, and simulated data have produced mixed results when applied to real data.A small automobile-insurance dataset is identified as the exception; publicly available spam email can provide an alternative, but requires feature selection or text processing.
  • Performance measures: Most supervised studies abandoned thresholded true positive rate and accuracy because false-positive and false-negative costs are unequal, uncertain, instance-dependent, and time-varying.Fraud departments instead assign monetary value to predictions through explicit cost or benefit models.

4. METHODS AND TECHNIQUES

The section reviews four major fraud-detection methods and their algorithms, spanning labelled, legal-only, and unlabelled data. It highlights evolving fraud patterns, comparative algorithm results, and the importance of operational constraints in technique selection.

  • 4.1 Overview: Fraud detection systems commonly use black lists and hard-coded rules to match new claims, applications, transactions, accounts, or sequences against known fraud indicators.Rules may require matching addresses and phone numbers or enforcing price and amount limits.
  • 4.1 Overview: Fraudsters continually introduce new or modified styles after their current modus operandi is uncovered, causing detection systems to generate false negatives.This adversarial evolution complicates the use of static detection patterns.
  • Supervised methods: Bayesian networks were more accurate and faster to train than neural networks, but slower when applied to new credit-transaction instances.The comparison used STAGE for Bayesian networks and backpropagation for neural networks.
  • Supervised methods: Stacking and bagging combined neural networks, naive Bayes, and C4.5 classifiers trained on minority-oversampled partitions to produce the best cost savings on automobile insurance claims.The approach used stacking to select base classifiers and bagging to combine their predictions.
  • Unsupervised and relational methods: Link analysis and graph mining captured an elusive fraudster by visually analysing twelve months of insurance claims and forming groups of tightly connected data.These techniques are described as under-rated in fraud-detection research despite broader use in security applications.
  • Practical considerations: In real-world fraud detection, technique selection depends more on operational requirements, resource constraints, and management commitment than on technical data issues.Only seven studies claimed implementation as actual fraud-detection systems, while few explicitly used temporal information and virtually none used spatial information.

5. OTHER ADVERSARIAL DOMAINS

Other adversarial domains offer fraud detection lessons through linked, spatial, temporal, textual, and multi-level data, along with alternative analytical methods. These domains also highlight challenges involving false positives, privacy, anomaly detection performance, and adversarial adaptation.

  • 5.1 Terrorist Detection: Terrorist detection research addresses skewed-data false positives and privacy concerns using fixed-size clustering, linked data structures, and a second-round classifier.Jensen et al. recommend generating true class labels through fixed-size clustering and combining scores from negative and positive entities’ normal distributions.
  • 5.1 Terrorist Detection: Spatial and spatio-temporal surveillance methods identify significant densities from emergency-department and retail medication data, supporting their application to fraud detection.Neill and Moore use Kulldorff’s spatial scan statistic and an overlap-kd tree with latitude, longitude, and zip-code data.
  • Financial Crime: Financial-crime systems combine Bayesian inference, link analysis, rule and sequence matching, visualisations, association rules, decision trees, text mining, regression, fuzzy matching, and uncertainty handling.These systems support money-laundering investigations, detect suspicious market behaviours and temporal relationships, and generate insider-trading breaks.
  • Financial Crime: Adversarial-domain research commonly uses unstructured text and web data, while new relevant data sources and learning algorithms can target earlier detection of insider trading.Examples include free-text documents, web pages, emails, SMS messages, and option-market signals before news releases.
  • Intrusion Detection: Intrusion-detection research spans host, network, and user data, but studies generally find anomaly detection performs worse than misuse detection.Approaches include clustering, hidden Markov models, multiple rule-learning methods, frequent episodes, attribute-oriented induction, k-means, and replicator neural networks.
  • Intrusion Detection: Game theory models strategic interaction between detection systems and adversaries, including mobile-network intrusions and cost-sensitive spam detection.The cited approaches address attacker-system interaction and automatic re-learning against a cost-sensitive adversary’s optimal strategy.

6. RELATED WORK

Prior work surveys fraud-detection techniques across credit-card, telecommunications, intrusion-detection, money-laundering, and related domains. This survey instead adopts a practical, data-oriented, performance-driven perspective and expands coverage of fraud types, methods, technical problems, criticisms, and future contributions.

  • Prior surveys: Earlier studies survey techniques across credit-card, telecommunications, intrusion-detection, and money-laundering domains, while Weatherford recommends neural-network and artificial-immune-system approaches.Weatherford specifically recommends backpropagation neural networks, recurrent neural networks, and artificial immune systems.
  • Survey perspective: This paper examines fraud detection from a practical, data-oriented, performance-driven perspective rather than the application-oriented or technique-oriented views of three other recent surveys.The distinction is explicitly framed against the typical perspectives of the other recent survey papers.
  • Scope and contribution: The survey defines underlying technical problems and covers more fraud types, methods, and techniques than the other survey papers.Its examples include internal fraud and various hybrid approaches.
  • Scope and contribution: It also presents criticisms of the current fraud-detection field and highlights possible future contributions from related domains to data mining-based fraud detection.These criticisms and future contributions complement the survey’s expanded technical coverage.

7. CONCLUSION & FUTURE WORK

The survey consolidates research on automated fraud detection by defining fraud, data, evaluation, and technical methods while identifying limitations. It proposes drawing on related fields, including counterterrorism, law enforcement monitoring, and text mining.

  • Contributions: The survey covers published fraud detection studies, defining the adversary, fraud types and subtypes, data characteristics, performance metrics, and technical methods.It also identifies limitations in existing fraud detection methods and techniques.
  • Future Work: Fraud detection can benefit from approaches developed in related fields.The paper specifically points to unsupervised approaches from counterterrorism and monitoring systems and text mining from law enforcement.
  • Future Work: The proposed cross-domain directions include unsupervised counterterrorism approaches, law-enforcement monitoring systems, and law-enforcement text mining.
Loading 1009.6119v1…