Source-linked AI summary

SR-Fraud: An Outcome-Supervised Reflective LLM Agent Framework for Non-Stationary Payment Fraud Detection

Xuwei Tan, Yao Ma, Xueru Zhang

arXiv:2609.27287v1cs.LGcs.CR

TL;DR

Payment fraud is non-stationary, and static classifiers can miss temporal behavior before retraining. SR-Fraud pairs a stateless, history-aware decision agent with offline reflection whose proposed rules are verified against matured outcomes. Reflection improves all reported detection metrics over the Sonnet 4.6 decision agent, with essentially the same block rate.

  • Problem

    Payment fraud patterns shift as adversaries adapt, while localized bursts can cause losses before models adapt and standard classifiers may miss temporal behavior without engineered features.

    Source: §1 · p. 1
  • Method

    SR-Fraud combines a stateless decision agent scoring a Hybrid Episodic Window with offline reflection that proposes boundary rules and verifies them against matured evidence.

    Source: §1 · pp. 1–2, §2.3 · p. 3
  • Results

    +5.4 precision, +7.3 recall, +6.2 F1, and +19.3 points dollar-weighted recall versus the Sonnet 4.6 decision agent, at essentially the same block rate (5.97% →6.08%).

    Source: §3.4.1 · p. 5
  • Takeaways & Limitations

    Chronological replay shows that behavioral context and verified reflection improve fraud capture over static and periodically retrained tree baselines, with an exact block-count comparison preserving the advantage at fixed customer friction.

    Source: §4 · p. 7

    The evaluation uses one non-releasable production stream, limiting independent replication and cross-platform generalization.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Real-time payment fraud detection is a non-stationary streaming prediction problem: adversaries adapt before supervised labels mature, and localized burst attacks can cause losses before retraining. Production systems typically rely on tabular classifiers and rules, which can struggle to capture these emerging sequential patterns before periodic retraining occurs. We present SR-Fraud, an outcome-supervised reflective LLM framework that decouples request-time decisions from offline adaptation. A frozen, stateless agent scores each transaction from a Hybrid Episodic Window to track behavioral shifts, while an offline reflection agent proposes boundary hypotheses from matured errors. A deterministic verifier then admits only supported hypotheses into an executable knowledge state. On a production payment-fraud benchmark, SR-Fraud improves all detection metrics over its frozen decision agent, obtains higher point estimates than static and periodically retrained CatBoost, and detects an emerging fraud burst.

1 Introduction

Payment fraud patterns shift as adversaries adapt, and localized bursts can cause losses before models adapt. SR-Fraud separates fast transaction decisions from offline, outcome-supervised adaptation to address the latency and adaptation gaps of inline LLMs.

1 Introduction

  • Localized burst attacks can cause losses before a model adapts, while adversaries’ changing patterns quickly decay from static training data.Payment fraud is framed as a non-stationary streaming problem.Source: §1 · p. 1
  • Standard GBDTs score transactions independently unless temporal behavior is engineered into aggregate features, and distribution shifts require periodic retraining and new features.Fraudulent transactions may look individually plausible but become recognizable against a user’s historical behavior.Source: §1 · p. 1
  • LLMs can use pre-trained knowledge and recent user history to interpret complex metadata and identify subtle anomalies without manual feature engineering or model training.The paper also notes that natural-language insights and rules can transfer across product lines.Source: §1 · p. 1
  • Inline LLM agents face a reasoning-budget gap from latency overhead and an adaptation gap because frozen models do not adjust to dynamic patterns or institution-specific policies.Source: §1 · p. 1
  • SR-Fraud uses a single-pass, stateless decision agent over a Hybrid Episodic Window and an offline reflection agent whose proposed rules are verified against matured labels.Verified rules form a compact, human-readable knowledge state that adjusts boundary cases.Source: §1 · pp. 1–2

2 Methodology

SR-Fraud separates request-time scoring from offline learning: a frozen agent uses transaction history and velocity context, while verified rules correct boundary decisions. Weekly reflection proposes updates from matured outcomes, and a deterministic harness tests them before they enter the knowledge state.

2.1 System Overview

  • The frozen decision agent assigns one of five bounded risk levels from a Hybrid Episodic Window, and the verified knowledge state deterministically adjusts boundary decisions.The system separates request-time inference from asynchronous offline learning.Source: §2.1 · p. 2

2.2 Transaction-Grounded Decision Agent

  • The decision agent treats scoring as contextual anomaly detection, serializing a user’s recent transactions and burst-and-velocity summary into structured text before assigning bounded risk.The Hybrid Episodic Window retains event-level context alongside retry, timing, amount, and channel-velocity signals.Source: §2.2 · p. 2
  • The expert knowledge state edits only boundary cases, escalating suspicious risk-3 transactions or releasing over-blocked risk-4 transactions while leaving the agent stateless and single-pass.Source: §2.2 · p. 2, §2.3 · p. 3

2.3 Outcome-Supervised Reflection

  • Each weekly reflection cycle analyzes matured errors and proposes candidate rules; a deterministic harness verifies candidates against matured data before updates enter the knowledge state.The LLM directs where to search, while evidence determines what enters the active policy.Source: §2.3 · p. 3
  • The active knowledge state pairs conjunctive escalation signatures with a Boolean release predicate, and its boundary layer executes in O(|S|) time under a fixed-size release predicate.The empty state exactly reproduces the decision agent.Source: §2.3 · p. 3
  • Because false blocks censor their counterfactual outcomes, reflection uses observable missed fraud for escalation and settled risk-3 traffic as a proxy to assess release safety.The proxy is validated against an oracle in Appendix D.7.Source: §2.3 · p. 3
  • The LLM proposes amount, qualifier, and release-band candidates, while the deterministic harness enumerates escalation signatures and controls their validation.Source: §2.3 · p. 4
  • One-sided Wilson confidence bounds and structural pre-tests constrain boundary updates to candidates with bounded evidence.Rules are admitted by lower-bound ranking and retired only when their marginal-precision upper bound falls below threshold and they fail the original train–validation gate.Source: §2.3 · p. 4

3 Experiments

Chronological replay evaluates SR-Fraud on a production payment-risk cohort and compares its contextual decision agent and reflection framework with tabular, sequence, and LLM baselines. Reflection improves detection over the decision agent, while behavioral context detects an emerging burst and post-training offers a separate deployment trade-off.

3.1 Data and Replay Protocol

  • The evaluation uses 16,140 transactions from a nine-week production stream, with a 5.04% fraud rate and a case–control sample of matured fraud and subsampled legitimate traffic.The company limits disclosure of broader-stream distribution and sampling statistics.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 Implementation Details

  • Weekly replay enforces a two-cycle maturation lag, forms each cycle’s policy only from outcomes through cycle j−2, and freezes each cycle after one scoring pass.The same propose–verify pipeline is applied to the preceding train–validation period before test replay.Source: §3.2 · p. 4

3.4.1 Overall Detection Performance

  • SR-Fraud improves over its decision agent by +5.4 precision, +7.3 recall, +6.2 F1, and +19.3 dollar-weighted recall points at essentially the same block rate.Against CatBoost-periodical at its 5% point, deltas are +0.3 precision, +8.1 recall, +3.8 F1, and +7.8 dollar recall; recall and dollar recall exclude zero.Source: Table 2 · p. 5, §3.4.1 · p. 5

3.3 Comparison Systems and Baselines

  • Comparisons include CatBoost and periodically retrained CatBoost, a supervised sequence model, direct and contextual LLM agents, verbal-memory Reflexion, and SR-Fraud.The CatBoost operating-point comparison uses a 5.0% block rate; Table 2 reports percentage-point differences from static CatBoost while block rates and latency remain absolute.Source: §3.3 · pp. 4–5, Table 2 · p. 5

3.4.2 Behavioral Context Drives LLM

  • The LLM advantage over both CatBoost variants widens with prior-history depth, as the LLM compares current events with raw episodic sequences rather than fixed history summaries.This widening gap motivates routing history-rich traffic to the contextual LLM layer.Source: §3.4.2 · pp. 5–6, Table 3 · p. 6

3.4.3 Detection of an Emerging Fraud Burst

  • On the burst, SR-Fraud exceeds static CatBoost at its native 5% point by +41.7 recall and +46.4 dollar-recall points.On the high-value subset, SR-Fraud’s deltas are +10.9 and +10.4, while the decision agent’s are negative; the two-cycle maturation wall means reflection could not learn from the week-8 burst itself.Source: Table 4 · p. 6, §3.4.3 · p. 6

3.4.4 Post-Training the Decision Agent

  • The post-trained self-hosted 9B decision agent attains similar detection performance while reducing request latency and, at sufficient sustained volume, modeled serving cost.It is evaluated without a symbolic knowledge state, complementing rather than combining with the reflection layer.Source: §3.4.4 · p. 6, Table 5 · p. 7

4 Conclusion

SR-Fraud combines a stateless contextual LLM with outcome-supervised symbolic reflection for non-stationary payment fraud detection. Chronological replay shows improved fraud capture over static and periodically retrained tree baselines while preserving an exact block-count comparison at fixed customer friction.

4 Conclusion

  • A deterministic verifier admits and retires boundary rules from matured evidence, preserving one request-time LLM call and an auditable knowledge state.Chronological replay shows behavioral context and verified reflection improve fraud capture over static and periodically retrained tree baselines.Source: §4 · p. 7

Limitations

The reported limitations concern verifier confidence control, evaluation reproducibility and generality, and the fidelity of replay and latency estimates to production.

Limitations

  • The verifier uses a heuristic one-sided Wilson bound without formal multiple-testing correction; stability across tested settings does not control adaptive comparisons.Re-verification of persisted mainrun candidates yielded a stable operating point, but the authors explicitly distinguish this sensitivity check from formal control.Source: Limitations · p. 7
  • Repeated full-stream decision replays were not conducted because of API and computational cost, so the entity bootstrap quantifies sampling uncertainty rather than repeated-run variability.Performance was stable across evaluated reflection and decision-backbone configurations.Source: Limitations · p. 7
  • The study uses one non-releasable production stream, limiting independent replication and cross-platform generalization; undisclosed traffic statistics also preclude end-to-end production-rate claims.The paper reports benchmark metrics as percentage-point differences relative to the corresponding CatBoost reference.Source: Limitations · p. 7
  • The fixed two-cycle maturation lag approximates the dominant historical pattern but abstracts away transaction-specific label-availability times.The fixed wall is used for a reproducible replay while omitting proprietary timing details.Source: Limitations · p. 7
  • Hosted latency percentiles are point-in-time request-level provider measurements, not production SLO guarantees or full-application latency.Provider load, routing, service changes, and network conditions can change the reported values.Source: Limitations · p. 7
  • Off-policy replay leaves later histories unchanged even when SR-Fraud would have blocked a transaction, although confirmed-fraud denylisting reduces subsequent same-entity fraud.Pre-confirmation attempts and legitimate counterfactual blocks could still reshape histories; policy-consistent or shadow evaluation is left to future work.Source: Limitations · p. 7

Ethics Statement

The paper describes safeguards and remaining risks in its payment-data workflow, then positions SR-Fraud among fraud-detection, tabular-LLM, and reflective-agent research.

Ethics Statement

  • Only vetted transaction attributes enter prompts and are serialized as data rather than instructions, reducing but not eliminating metadata, re-identification, or prompt-injection risk.Production disputes receive human analyst review, and reflection-loop knowledge updates can pass an optional human-review checkpoint; reported experiments run the loop automatically.Source: Ethics Statement · p. 7

A Related Work

  • Prior payment-fraud work spans data-mining classifiers, temporal features, drift adaptation (Gama et al., 2014; Dal Pozzolo et al., 2015), and sequence or streaming systems (Carcillo et al., 2018).TitAnt presents an online real-time transaction-fraud system at Ant Financial (Cao et al., 2019); SR-Fraud retains a tabular first pass and adds contextual LLM reasoning and verified symbolic adaptation.Source: A Related Work · p. 10
  • TabLLM (Hegselmann et al., 2023), UniPredict (Wang et al., 2023), and TabPFN (Hollmann et al., 2023) explore foundation models for tabular data, while finance and fraud work studies financial text or structured-data applications.SR-Fraud instead uses a frozen request-time scorer and outcome-supervised symbolic updates without online parameter optimization.Source: A Related Work · p. 10
  • Reflective-agent approaches include verbal memory or iterative feedback in Reflexion and Self-Refine (Madaan et al., 2023), alongside search and other feedback-based systems.SR-Fraud proposes hypotheses from weekly error batches and admits boundary rules only after deterministic verification against matured labels; its replay requires no human guidance.Source: A Related Work · p. 10

B SR-Fraud Implementation Details

The implementation separates weekly, maturation-constrained offline updates from single-pass request-time scoring, with a verifier controlling the rules that enter the evolving state.

B.1 Verification and Replay Configuration

  • Fixed verification settings include a 0.020 legitimate-traffic rarity ceiling, a 0.30 conditional-precision floor, and Wilson-bound gates for admission, corroboration, retirement, and release.The configuration also requires minimum matched-row and voting-cycle counts and admits up to five qualifying escalation signatures per cycle.Source: §B.1 · p. 10
  • Weekly cycles impose a two-cycle maturation lag: cycle j uses outcomes only through j−2, and each cycle is scored once and frozen rather than rescored under later state.Reflection runs offline, outside the request path, and the digest builder is restricted by the maturation wall enforced by the harness.Source: §B.1 · p. 10
  • Before cycle j, the workflow digests matured data, has the reflection agent propose hypotheses, enumerates candidates, verifies them, then selects, retires, or admits rules before scoring the cycle.The same propose–verify procedure constructs an initial state from training and validation before test replay; experiments deploy the resulting state directly.Source: §B.1 · p. 11

B.2 Velocity Summary Computation

  • The velocity block summarizes recent history using minimum and median inter-arrival times, counts in the last 10 and 30 minutes, field diversities, and historical amounts.Timestamps are parsed to UTC, and tail-window counts are computed from sorted timestamps.Source: §B.2 · p. 11

B.3 Decision Agent Prompt

  • The decision prompt treats up to 15 prior transactions and a computed burst/velocity block as context, then requests one of five bounded risk levels for the current transaction.The prompt warns that recent success does not guarantee trust and defines fraud as a likely dispute resulting in financial loss.Source: §B.3 · p. 11

B.4 Reflection Agent Prompt

  • The offline reflection agent proposes feature atoms and release bands from matured operating statistics and error profiles; only the deterministic harness verifies and admits or retires rules.The rationale is retained for audit but never executed, and atoms outside the fixed vocabulary are dropped.Source: §B.4 · p. 11

C.1 Benchmark Construction

The benchmark uses a fixed production-derived cohort and compares supervised, sequence, LLM, and reflective baselines under specified training and replay procedures.

C.1 Benchmark Construction

  • The evaluation cohort contains 16,140 upstream-allowed transactions with 813 matured fraud cases, but its 5.04% fraud share is not production prevalence because legitimate traffic was subsampled.All systems score the same cohort; precision, F1, and block rate describe its operating point, while results are interpreted conditionally rather than as end-to-end production rates.Source: §C.1 · p. 12
  • Earlier, disjoint periods support model training and operating-point selection; evaluation transactions are timestamp-ordered into nine weekly cycles, with recurring entities used only for prior-history features.Transaction-level splits have no overlap by transaction ID or entity-hash/timestamp pair; the study also reports aggregate cohort statistics and replay and verification procedures because records cannot be released.Source: §C.1 · p. 12

C.2.1 Supervised CatBoost

  • CatBoost uses more than 100 features including rolling user-history aggregates, native categorical handling, positive-class weighting, validation early stopping, and a 5.0% block-rate operating point.Fifty Optuna trials select the highest validation PR-AUC; CatBoost-periodical retrains monthly while respecting the same two-cycle maturation lag.Source: §C.2.1 · p. 12

C.2.2 Supervised Sequence Model (Seq-GRU)

  • Seq-GRU receives the current transaction and up to 10 strictly earlier events, using CatBoost features without rolling aggregates to test whether gains reflect history access or reasoning over it.A three-layer GRU produces continuous scores evaluated at a 5.0% block-rate operating point.Source: §C.2.2 · p. 12

C.2.3 LLM Comparison Systems

  • The Direct LLM is a no-history control, while Decision Agent-only adds the Hybrid Episodic Window but keeps the knowledge state empty and performs no outcome-supervised adaptation.Both comparisons use the same online decision procedure for assessing the contribution of history and adaptation.Source: §C.2.3 · pp. 12–13

C.2.4 Reflexion

  • Reflexion maps an episode to a week of scoring, label maturation, and memory writing, with the same weekly cadence and maturation wall as SR-Fraud but unverified free-text updates.Its memory receives the same evidence, yet candidate rules lack SR-Fraud’s rarity screen, fraud-enrichment test, and matured-precision promotion gate; memory writing adds no live-path calls.Source: §C.2.4 · p. 13

D Additional Experiments

  • The supplementary experiments cover mechanism checks, verifier robustness, and deployment-efficiency comparisons; Table 7 reports paired SR-Fraud-minus-baseline differences with entity-clustered bootstrap 95% confidence intervals.The evaluated model configurations include claude-opus-4-5, claude-sonnet-4-6, and gpt-5.4.Source: D Additional Experiments · p. 13, Table 7 · p. 13, §C.3 · p. 13

D.1 Statistical Reliability

Entity-clustered paired bootstrap results support an overall reflection gain over the frozen decision agent, while the LLM’s advantage over CatBoost-periodical is concentrated in deeper-history cohorts. A history-content ablation further indicates that the agent benefits from the correct entity’s behavioral history, not merely from having any history block.

D.1 Statistical Reliability

  • All four full-benchmark reflection gains have confidence intervals above zero, including +6.2 F1 [3.6, 9.0] and +19.3 Recall$ [14.3, 24.5].The paired entity-clustered bootstrap resampled entities for both systems (B=2000).Source: §D.1 · p. 13
  • Within smaller history cohorts, precision, recall, and F1 intervals for reflection gains include zero; dollar-recall gains exclude zero for ≥1 and ≥2, but not ≥3.The reported dollar-recall gains are +12.4 [7.0, 18.3] for ≥1, +7.9 [0.3, 15.3] for ≥2, and +10.4 [−0.4, 20.8] for ≥3.Source: §D.1 · p. 13
  • From the ≥1 history cohort onward, recall, F1, and dollar-recall intervals exclude zero against CatBoost-periodical, reaching +29.9 Recall$ [14.5, 44.8] at ≥3 prior transactions.Full-benchmark precision and F1 are indistinguishable from CatBoost-periodical; its recall and dollar recall are significantly lower than SR-Fraud’s.Source: §D.1 · pp. 13–14

D.2 History-Content Utilization

  • On the ≥3-prior-history cohort, correct history improves precision, recall, F1, and dollar-weighted recall over CatBoost-periodical by +9.3, +24.7, +17.8, and +19.5 points.Wrong-entity history yields a 60.6% block rate, while removing history yields negative recall and dollar-recall deltas; the pattern persists across donor seeds and amount-matched donors.Source: §D.2 · p. 14

D.3 Proposer Ablation

The ablations examine proposal strategies, operating points, matched-budget baselines, backbone roles, rejection inference, verification sensitivity, and online updates. Together, they report the strongest proposer point estimates for SR-Fraud, identify where its gains arise, and describe boundaries on their interpretation.

D.3 Proposer Ablation

  • SR-Fraud has the strongest proposer F1 and dollar-recall point estimates (+8.0 and +11.7 points), while raw-space search is infeasible and beam rules cover under 0.1% of benchmark traffic.The six classical-miner configurations reach at most +5.4 F1 and +9.6 dollar-recall points; raw enumeration entails 2.4 × 10^7 candidates per cycle, and beam search loses 31.0 F1 and 48.2 dollar-recall points versus static CatBoost. The selected feature subset improves both measures for every classical miner.Source: §D.3 · pp. 14–15, Table 9 · p. 15

D.4 Reflexion Operating-Point Sensitivity

  • Reflexion exceeds the target block budget at risk ≥4 across all tested backbones, and raising the threshold to risk ≥5 does not consistently fix it.Realized block rates range from 14.8% to 41.4%; the cross-backbone comparison diagnoses Reflexion and is not a measured SR-Fraud comparison across every backbone.Source: §D.4 · p. 15

D.5 CatBoost at SR-Fraud’s Block Budget

  • At a common block count, SR-Fraud’s precision, recall, and F1 advantages over static CatBoost are significant, while none of its differences from CatBoost-periodical is statistically significant.Against static CatBoost, the differences are +6.8 precision [2.1, 11.5], +8.2 recall [1.3, 14.6], +7.5 F1 [2.1, 12.7], and +7.3 dollar recall [−1.1, 15.2].Source: §D.5 · pp. 15–16, Table 12 · p. 16

D.6 Reflection- and Decision-Backbone Combinations

  • The decision backbone sets the operating point: Sonnet yields the strongest F1 and dollar-recall for every reflector, while the reflector matters less within a column.With Sonnet as decision agent, F1 deltas are +8.0, +8.7, and +8.1 points for Sonnet, Opus, and GPT reflectors, respectively.Source: §D.6 · p. 16, Table 14 · p. 16

D.7 Rejection-Inference Validation

  • Proxy and oracle rejection-inference labels select the same release family, with proxy recall and dollar-recall deltas of +12.3 and +11.7 versus static CatBoost.Oracle-label deltas are +15.0 and +14.2; the block rate changes from 6.08% to 6.55%, and release remains restricted to the matching risk-4 slice.Source: §D.7 · p. 17

D.8 Verification Confidence Sensitivity

  • The verifier’s operating point remains stable when persisted candidates are reverified at z ∈{1.64, 1.96, 2.58}, with the remaining pipeline fixed.The reported run uses z=1.0; the additional settings are sensitivity checks rather than family-wise confidence guarantees.Source: §D.8 · p. 17, Table 15 · p. 17

D.9 Candidate Admissions and Multiplicity

  • Matured evidence delays corroboration of r=3 signatures until cycle 4; GPT saturates its five-signature cycle cap, while Sonnet is most selective and the only configuration to retire rules.Holding the pre-replay state fixed reduces F1 by 0.4/2.4/1.7 points and dollar-recall by 4.3/11.2/5.9 points for Sonnet/Opus/GPT.Source: §D.9 · p. 17

D.10 Online Reflection vs. a Static State

  • Online reflection improves dollar-recall for every tested backbone compared with holding the pre-replay knowledge state fixed.The decision agent, initialization procedure, verifier, and rejection-inference release are otherwise unchanged.Source: §D.10 · p. 17

D.11 Request-Time Latency Analysis

SR-Fraud keeps reasoning and reflection off the request path: each transaction receives a single-pass decision, while policy adaptation is handled offline. The reported comparison shows much higher latency for inference-time chain-of-thought than for single-pass SR-Fraud and Reflexion.

D.11 Request-Time Latency Analysis

  • Single-pass SR-Fraud and Reflexion remain near 1.3 s at p50, while chain-of-thought reaches 47.3 s and greatly increases output length.SR-Fraud amortizes reasoning into the offline reflection cycle without adding request-time reasoning calls; the distinction from Reflexion is verification quality rather than latency.Source: §D.11 · p. 17, Table 19 · p. 18

D.12 Post-Training and Deployment Analysis

The deployment analysis reports that newer closed-model releases do not consistently improve fraud detection and describes post-training as a complementary way to adapt a self-hosted decision model. It also discusses the cost and latency trade-offs of an additive LLM deployment and the option of offline human review.

D.12 Post-Training and Deployment Analysis

  • Across three model families, newer closed releases fail to improve fraud detection; Opus 5 and Sonnet 5 regress on all four metrics, while GPT-5.6 raises precision only with substantially greater conservatism.The discussion motivates adapting a self-hosted model whose training and deployment remain under system control.Source: §D.12 · p. 18, Table 20 · p. 18
  • Post-training lets the smaller self-hosted model reach a detection profile similar to the commercial SR-Fraud configuration, but new fraud patterns require another training and deployment cycle.The paper frames post-training as a strong base policy and reflection as an auditable adaptation channel between model refreshes.Source: §D.12 · pp. 18–19
  • The self-hosted agent’s median latency is 0.74 s and p90 is 0.94 s; at 10,000 calls per day, every listed self-hosting option costs under a third of Sonnet’s and under a fifth of Opus’s API price.The measured setup serves the 9B agent on one GPU; self-hosting trades per-token API charges for provisioned GPU capacity.Source: §D.12 · p. 19, Table 21 · p. 19

E Additional Discussion

  • An optional human checkpoint can review each knowledge-state changelog before deployment, with weekly offline review leaving request-time latency unaffected.The discussion also cites upstream-model latency and per-request LLM cost as reasons not to replace the tabular classifier end to end.Source: E Additional Discussion · p. 19
Loading 2609.27287v1…