PaperlayerView exact revision on arXiv

Source-linked AI summary

FraudBench: Protocol-Sensitive Benchmarking of Adversarial Robustness for Financial Risk Assessment

Xitong Zeng, Zhaoge Bi, Yitian Yang, Huaming Chen, Quan Z. Sheng

arXiv:2608.24551v1cs.LGcs.AI

TL;DR

Financial fraud and credit-risk robustness is difficult to assess because tabular data impose domain constraints, class imbalance, and asymmetric attacker capabilities. FraudBench compares matched unconstrained, post-hoc filtering, and deployment-aware attack protocols across financial datasets, models, attacks, and defences. Robustness conclusions vary substantially with protocol, motivating joint reporting of predictive degradation and attack feasibility.

  • Problem

    Financial fraud robustness lacks evaluation that jointly reflects domain validity, severe class imbalance, and asymmetric attacker capability.

  • Method

    FraudBench evaluates matched dataset–model–attack–defence settings under unconstrained, post-hoc filtering, and deployment-aware constraint-integrated attack protocols.

  • Results

    Robustness conclusions are highly protocol-sensitive, including changes in feasible attack counts, feasibility–capability effects, and model-family rankings.

  • Takeaways & Limitations

    Fraud robustness evaluation should report predictive degradation and attack feasibility jointly while incorporating domain constraints and attacker capability into attack generation.

  • Takeaways & Limitations

    The main experiments use only three seeds, the processed-space L∞ budget is not monetary or attacker-cost-aware, and CAPGD does not apply directly to XGBoost.

Abstract

from arXiv · show

Machine learning models are widely used in financial fraud and credit-risk detection, yet their adversarial robustness remains difficult to evaluate because financial tabular data involve domain-specific constraints, severe class imbalance, and asymmetric attacker capability. We argue that, in this setting, robustness is not only an attribute of the model, but also an attribute of the evaluation protocol. Different ways of enforcing constraints and capability can lead to substantially different robustness conclusions. This paper presents FraudBench, a protocol-sensitive benchmark for adversarial robustness evaluation in financial fraud and credit-risk detection. Rather than treating domain constraints as post-hoc validity checks, FraudBench evaluates the same dataset--model--attack--defence setting under three matched protocols: unconstrained attacks, post-hoc feasibility filtering, and deployment-aware constraint-integrated attacks. FraudBench covers four public financial datasets, and evaluates neural, tree-based, and ensemble models using three attack settings. Our results show that robustness conclusions are highly protocol-sensitive. On Lending Club Loan Data under the white-box setting, post-hoc filtering leaves only 3.7 feasible-flipped examples on average, whereas in-attack projection with attacker mutability masking produces 2,832.3 feasible-flipped examples under the same perturbation budget. The results on IEEE-CIS further show that feasibility and attacker capability are separate axes, while black-box evaluation shows that protocol choice can alter model-family rankings. These findings suggest that fraud robustness evaluation should report predictive degradation and attack feasibility jointly, and should incorporate domain constraints into attack generation rather than treating them as post-processing checks.

I. INTRODUCTION

Financial fraud robustness is difficult to evaluate because tabular records have domain constraints, severe class imbalance, and asymmetric attacker capabilities. FraudBench therefore treats evaluation protocol as a first-class variable and compares matched attack protocols, datasets, models, and defences.

  • Financial fraud screening relies on machine-learning models because fraud is rare and manual review of every transaction is economically infeasible.Global card payment fraud losses reached $33.41 billion in 2024.
  • Severe class imbalance makes accuracy misleading, motivating robustness evaluation that reflects rare fraudulent cases and structured-data model choices.Fraudulent cases can represent far below one percent of records, while tree-based models remain strong tabular baselines.
  • Financial tabular attacks must preserve operational validity, including feature ranges, categorical structure, mutability, and cross-feature relationships.Invalid perturbed records may be rejected before model inference.
  • Existing benchmarks leave unresolved whether attackers constrained by realistic mutability and domain validity can evade financial detectors.General robustness benchmarks emphasize image-domain threat models, while fraud benchmarks often focus on natural predictive performance or privacy-aware evaluation.
  • FraudBench evaluates the same dataset–model–attack–defence setting under unconstrained attacks, post-hoc filtering, and deployment-aware constraint-integrated generation.The benchmark covers four public datasets and production-relevant neural, tree-based, and ensemble model families with white-box and black-box attacks.
  • 6.0?

II. RELATED WORK

Prior work establishes constrained tabular attacks and strong robustness-evaluation protocols, but financial fraud robustness remains fragmented across datasets, models, attacks, defences, and metrics. FraudBench addresses this gap by comparing robustness conclusions across matched evaluation protocols while broadening coverage beyond neural models and accuracy alone.

  • Prior approaches: Financial tabular constraints include immutable features, domain validity, relational dependencies, and consistency requirements.Examples include positive transaction amounts, valid one-hot categories, and arithmetic relationships among loan fields.
  • Prior approaches: Existing constrained-tabular attacks use gradient updates, constraint repair, evolutionary optimisation, or projected-gradient extensions.The cited methods include FENCE, MOEVA, CAPGD, and CAA.
  • FraudBench contribution: FraudBench measures how robustness conclusions change across evaluation protocols rather than treating feasibility as a post-hoc validity check.The benchmark builds on constrained-tabular evaluation work while focusing on protocol sensitivity.
  • Motivation: Financial fraud detection faces severe class imbalance, making accuracy misleading when fraudulent cases are rare.PR-AUC is identified as a more appropriate rare-class-sensitive metric.
  • Motivation: Tree-based models remain strong tabular baselines, so robustness benchmarks should evaluate them alongside neural and ensemble models.XGBoost and LightGBM are cited as widely used practical risk-modelling baselines.
  • Research gap: Prior fraud-robustness studies show vulnerability to adaptive manipulation, but remain fragmented across datasets, model families, attacks, defences, and metrics.The literature includes evasion, reinforcement-learning, imbalanced-tabular, and transferable attacks.

III. FRAUDBENCH DESIGN

FraudBench is a unified, config-driven benchmark for evaluating financial tabular robustness across datasets, model families, attacks, defences, and matched protocols. Its design combines four public datasets with differing task types, class prevalences, feature semantics, and constraint regimes.

  • Benchmark design: FraudBench specifies a dataset, model family, attack, defence configuration, perturbation budget, and random seed within one unified experiment design.The pipeline is config-driven and grounds robustness conclusions in the evaluation protocol rather than introducing a new detector.
  • Experimental coverage: The benchmark covers four public financial datasets, three model families, white-box and black-box attacks, four defence configurations, and three random seeds.These axes enable matched protocol comparisons within the same dataset–model–attack setting.
  • Datasets: The datasets span credit-card fraud, online transaction fraud, loan default, and simulated transaction fraud.LCLD represents credit risk/default and provides financial-domain constraints distinguishing borrower-controlled from institution-controlled fields.
  • Preprocessing and splits: FraudBench treats the data as static binary classification tasks, using a stratified 10% subsample and a stratified 60/20/20 train/validation/test split.The cached split is reused across model, attack, and defence configurations within each seed.
  • Preprocessing and splits: Preprocessing standardises numerical variables, one-hot encodes categorical variables, and imputes missing values according to feature type.These transformations define the processed tabular inputs used by the benchmark.
  • Constraint regimes: Constraint regimes differ across datasets: CCFD lacks meaningful semantic constraints, while IEEE-CIS includes one-hot validity and non-negativity or positivity constraints.CCFD serves as a negative-control setting for testing whether post-hoc filtering changes protocol conclusions.

C. Models, Attacks, and Defences

FraudBench evaluates neural, tree-based, and ensemble models with white-box and black-box attacks under matched protocol conditions. Its defence settings include no defence, adversarial training, input validation, and heterogeneous ensembling.

  • Models: FraudBench evaluates XGBoost, a two-hidden-layer MLP, and a soft-voting ensemble combining logistic regression, XGBoost, and the MLP.The MLP uses ReLU activations, class-weighted binary cross-entropy, and Adam optimisation.
  • Attacks: CAPGD is the primary white-box attack, while Square Attack and HopSkipJump provide black-box evaluation.CAPGD uses an L∞ budget of ϵ = 0.1 and ten iterations unless otherwise specified; tree-model robustness is read primarily from black-box attacks.
  • Defences: FraudBench evaluates no defence, adversarial training, input validation, and heterogeneous ensembling.No defence is the baseline, adversarial training uses CAPGD-generated examples, and input validation clips numerical values outside a three-standard-deviation range.
  • Evaluation design: The protocol comparison holds dataset split, preprocessing, model, attack budget, and random seed fixed while varying constraint and capability handling.This matched design isolates the effect of the evaluation protocol itself.
  • Protocol A: Protocol A generates unconstrained adversarial samples and measures predictive degradation under the raw attack.It is useful as a stress test, but unconstrained perturbations can create invalid categorical or loan records.

B. Protocol B: Post-Hoc Filtering

Protocol B reuses unconstrained adversarial samples and applies feasibility filtering afterward, measuring which attacks survive validity checks. This estimates attack filterability rather than whether a constraint-aware attacker can directly generate feasible attacks.

  • Protocol B: Protocol B applies a feasibility checker to the adversarial samples generated by Protocol A.The checker retains records satisfying all implemented domain constraints.
  • Protocol B: Predictive metrics under Protocol B are computed only on the feasible adversarial subset.This changes the evaluated sample set before metric computation.
  • Interpretation: Protocol B measures how many unconstrained adversarial examples survive validity filtering, not direct generation by a constraint-aware attacker.A low feasible-flipped count may therefore reflect attack-generation inefficiency rather than genuine model robustness.
  • Contrast with Protocol C: Protocol C instead integrates deployment constraints into the attack loop through in-attack projection and mutability masking.Projection enforces feasibility, whereas masking enforces attacker capability.
  • C1 comparison: C1 projects intermediate samples after each attack step and repairs selected feature groups into the feasible set.On LCLD, projection enforces the instalment amortisation relationship; on IEEE-CIS and Sparkov, it restores one-hot validity.

2) C2: In-Attack Projection with Mutability Masking:

C2 extends in-attack projection with a mutability mask, combining domain feasibility with an explicit model of which processed features the attacker may modify.

  • Mutability masking: C2 masks attack updates before sample updates so immutable features remain unchanged throughout the attack.The mask specifies allowed processed features and is derived from a raw-feature threat model after preprocessing.
  • Mutability masking: The mapping preserves raw-feature mutability across derived processed dimensions, including one-hot categorical blocks.Mutable raw categorical features expose all corresponding one-hot dimensions, while immutable features freeze them.

D. Protocol Execution

FraudBench executes matched attack protocols and reports predictive degradation together with feasibility-aware outcomes. These metrics distinguish ranking degradation from the number of plausible successful attacks.

  • Metrics: FraudBench reports clean PR-AUC, robust PR-AUC, aggregate feasibility, feasible-flipped counts, and filtered success rate.PR-AUC is used because severe class imbalance makes accuracy unreliable as a ranking measure.
  • Metrics: A feasible-flipped example both changes the prediction and remains valid under all domain constraints.The filtered success rate is the proportion of successful prediction flips that also satisfy those constraints.
  • Pipeline: The execution pipeline takes a test sample, label, trained model, attack operator, constraint checker, projection operator, mutability mask, and protocol as inputs.It outputs an adversarial sample together with prediction-flip and feasibility indicators.
  • Protocol execution: Figure 2 contrasts unconstrained attacks, post-hoc filtering, in-attack projection, and projection with mutability masking.Protocol A returns the unconstrained sample; B filters that sample; C1 projects during attack; C2 additionally masks updates.
  • Interpretation: PR-AUC measures ranking degradation under attack, whereas feasible-flipped count measures successful attacks that remain plausible under business rules and capability assumptions.A model can show robust-PR-AUC degradation while most attacks are infeasible, or appear safe after filtering while remaining vulnerable to feasible attacks.

V. EXPERIMENTAL SETUP

FraudBench evaluates neural robustness under a fixed CAPGD stress test and compares unconstrained attacks, post-hoc filtering, and deployment-aware protocols. The results show that feasibility measurements can diverge sharply from predictive degradation and depend on domain constraints.

  • Attack and evaluation design: CAPGD uses an L∞ perturbation budget of ϵ = 0.1 in processed feature space and ten attack iterations.The budget is framed as a first-order robustness stress test rather than a complete economic attack-cost model.
  • Attack and evaluation design: Experiments are replicated over seeds 42, 123, and 456, with registries recording predictive, feasibility, constraint, and runtime outcomes.The table-level artifacts include CAPGD and Square Attack registries sufficient to verify reported numerical aggregates.
  • Metrics: PR-AUC is the primary predictive metric because accuracy can remain high when rare fraudulent records are poorly ranked.PR-AUC still requires feasibility metrics because it does not indicate whether adversarial records remain valid.
  • Protocol comparison: Protocol B removes infeasible adversarial samples before evaluation, whereas C1 and C2 integrate projection, with C2 additionally applying mutability masking.Protocol B therefore measures filterability rather than intrinsic robustness.
  • Protocol comparison: On CCFD, Protocols A and B are identical because no semantic constraint catalogue is defined.This functions as a negative-control case for post-hoc feasibility filtering.
  • Protocol comparison: 2,832.3 feasible-flipped examples are produced by C2 on LCLD, compared with 3.7 retained after post-hoc filtering.C1 produces 2,071.0 feasible-flipped examples by integrating the instalment formula during attack generation.
  • Protocol comparison: 93.7 feasible-flipped examples result from IEEE-CIS one-hot projection, while mutability masking reduces the count to 4.3.On Sparkov, unconstrained attacks produce zero feasible-flipped examples, whereas C1 and C2 generate feasible attacks directly.

C. Protocol-Dependent Model-Family Rankings

Black-box Square Attack results show that protocol choice changes robustness conclusions across model families, not merely reported metric values. Protocol B can reorder which model appears most robust relative to Protocol A.

  • Black-box evaluation: Square Attack is the primary cross-model black-box attack because it does not require gradient access.This makes it especially relevant for XGBoost and the heterogeneous ensemble.
  • Protocol-dependent rankings: On IEEE-CIS, MLP ranks first under Protocol A, whereas XGBoost ranks first under Protocol B.The comparison directly demonstrates protocol-induced model-family ranking changes.
  • Protocol-dependent rankings: Figure 3 compares filtered success rate and robust PR-AUC, showing direct feasible attack generation in C1 and C2 against post-hoc retention in Protocol B.On IEEE-CIS, C1 benefits from one-hot projection, while C2 reflects mutability-mask suppression.
  • Black-box evaluation: Figure 4 reports robust PR-AUC under Protocols A and B with seed-variation error bars, highlighting large protocol-dependent shifts for IEEE-CIS, LCLD, and Sparkov.CCFD shows almost identical results because no semantic constraint catalogue is defined.
  • Protocol-dependent rankings: On LCLD, the best model changes from XGBoost under Protocol A to the ensemble under Protocol B.On Sparkov, XGBoost remains first but the lower-rank ordering changes.
  • Evaluation boundary: HopSkipJump is only a partial XGBoost cross-check and is excluded from main model-family rankings because its coverage is incomplete and computational cost is high.Square Attack remains the primary black-box comparison.

D. Deployment-Aware Defences

Deployment-aware defence comparisons align each dataset with the protocol used for adversarial evaluation. Adversarial training is strongest on constrained datasets, while z-score validation provides no reliable benefit and ensemble performance varies by dataset.

  • Evaluation protocol: IEEE-CIS, LCLD, and Sparkov use Protocol C2, while CCFD uses Protocol A because no semantic constraints are defined.Defence comparisons are therefore aligned with deployment-aware adversarial evaluation where constraints exist.
  • Defence results: Adversarial training is the strongest neural-model defence on constrained datasets under Protocol C2.It improves robust PR-AUC on IEEE-CIS, LCLD, and Sparkov.
  • Defence results: Z-score input validation provides no reliable robust-PR-AUC benefit and is consistently weaker than adversarial training on constrained datasets.The result applies to the tested setting.
  • Defence results: The ensemble achieves the highest mean robust PR-AUC on CCFD but is weak on constrained datasets, especially IEEE-CIS and Sparkov.The ensemble column is a cross-model comparison rather than a neural-model defence.
  • Interpretation: Protocol B can leave only a few valid attacks because it filters outputs from an unconstrained attack rather than testing a constraint-respecting attacker.This distinction is central to interpreting deployment-aware defence robustness.
  • Feasibility and capability: IEEE-CIS projection raises feasible-flipped counts from 0.0 to 93.7, while mutability masking reduces them to 4.3.The result separates record feasibility from the attacker’s access to predictive dimensions.
  • Feasibility and capability: Constraint-aware attacks remain dangerous when attackers can modify predictive features, but capability constraints can reduce success when predictive signals are immutable or institution-controlled.This frames attacker mutability as a separate axis from domain validity.

C. Dual Reporting is Necessary

FraudBench shows that robustness interpretation depends on reporting predictive degradation together with attack feasibility under matched, protocol-sensitive evaluation. Its application-specific scope and methodological limits define how its results should be used.

  • Feasible-flipped count measures attacks that simultaneously fool the model and satisfy all domain constraints.PR-AUC measures ranking degradation, while feasibility measures constraint validity; neither alone captures their intersection.
  • FraudBench recommends jointly reporting clean PRAUC, robust PR-AUC, aggregate feasibility, feasible-flipped count, and filtered success rate.This combination separates predictive degradation from attack feasibility and filterability.
  • FraudBench addresses financial fraud and credit-risk detection rather than replacing TabularBench’s broader tabular deep-learning coverage.Its narrower scope emphasizes severe class imbalance, production-relevant tree models, domain constraints, and feasibility-aware attacker success.
  • 3.7 to 2,832.3 feasible-flipped count on LCLD shows that post-hoc filtering and in-attack valid-record generation yield sharply different safety interpretations.Under Protocol B, most generated records may be invalid; under Protocol C, valid records are generated during the attack.
  • Three seeds, a processed-space L∞ budget, and CAPGD’s inapplicability to XGBoost constrain interpretation of the experiments.Tree-model robustness should mainly be read from black-box attacks, while broader comparisons, datasets, and projection operators remain future work.
  • FraudBench is intended for defensive evaluation, model auditing, and benchmark comparison, not operational guidance for committing fraud.
Loading 2608.24551v1…