PaperlayerView exact revision on arXiv

Source-linked AI summary

AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis

Natapong Nitarach, Phume Ngampornsukswadi, Pittawat Taveekitworachai, Surapon Nonesung, Warit Sirichotedumrong, Kunat Pipatanakul

arXiv:2610.13627v1cs.CL

TL;DR

AMLworld’s smallest test splits require over a million graph prompts per full evaluation, while balanced samples alter unweighted precision and F1. AMLworld-Compact retains every illicit case and samples benign cases by supervised-score strata, using Horvitz–Thompson weights to estimate full-test performance. Held-out booster evaluations show lower F1 point error than uniform benign sampling, while evaluated LLMs have precision near the full-test illicit prevalence.

  • Problem

    Evaluating AMLworld’s smallest test partitions requires over a million graph prompts per pass, while balanced samples change unweighted precision and F1.

    Source: §1 · pp. 1–2

  • Method

    HT-Coreset retains every illicit test edge, samples benign edges within supervised-score strata, and uses known inclusion probabilities to estimate full-test confusion counts.

    Source: §1 · p. 2, §3.3 · p. 3

  • Results

    Hard-negative sampling has lower F1 point error than uniform benign sampling in the held-out booster study, while seven open-weight LLMs have weighted precision near full-test illicit prevalence.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The evaluated prompts yield high recall alongside precision near full-test illicit prevalence, and lexical-check passes do not establish correctness because they also pass for over half of incorrect binary verdicts.

    Source: §6 · p. 8

    The main LLM evaluation covers seven open-weight models on two small synthetic splits with one graph-to-text pipeline, and its evaluated inputs do not consistently identify and include the target transaction.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Evaluating large language models (LLMs) on AMLworld's smallest test splits is costly because each of over a million targets requires a separate graph prompt. Sampling reduces this cost, but small balanced subsets change unweighted precision and F1. We introduce AMLworld-Compact, an evaluation set that retains every illicit test transaction and samples benign transactions from supervised-score strata. Its HT-Coreset protocol reduces evaluation targets by 271x and 611x on HI-Small and LI-Small. Horvitz-Thompson weights give unbiased full-test confusion-count estimates for fixed predictions; recall is exact, while precision and F1 are ratio estimates. In a held-out study using Graph Feature Preprocessor (GFP) features, one boosted-tree family defines strata for evaluating the other over 500 draws per configuration. At the released sizes, F1 RMSE is 0.00-2.58 percentage points versus 4.44-7.41 for uniform sampling with HT weights; conservative intervals remain broad. Under the evaluated graph prompts, seven open-weight LLMs have weighted precision near the predict-all-illicit baseline. In an outcome-stratified audit of 1,000 HI-Small few-shot outputs from one run per model, lexical checks pass for 84.5% of correct and 56.2% of incorrect binary verdicts, so these markers do not establish correctness. We release the evaluation sets, construction code, and audit tools at https://huggingface.co/datasets/typhoon-ai/AMLworldCompactEval and https://github.com/nat-nischw/AMLworld-Compact.

1 Introduction

Evaluating AMLworld’s smallest test splits requires over a million graph prompts per pass, while balanced subsets alter unweighted precision and F1. AMLworld-Compact retains every illicit target, samples benign targets by supervised-score strata, and uses inclusion weights to estimate full-test performance.

1 Introduction

  • Each smallest AMLworld test partition contains over a million transactions, so evaluating every target separately requires over a million graph prompts per pass.Comparing models and prompt variants multiplies that workload.Source: §1 · p. 1
  • Balanced subsets change unweighted precision and F1, while class-proportional samples may leave too few illicit cases for recall and typology analysis.The goal is to reduce model calls while retaining rare cases for diagnosis and estimating detection at full-test prevalence.Source: §1 · pp. 1–2
  • HT-Coreset retains every illicit test edge, samples benign edges within supervised-score strata, and uses known inclusion probabilities for Horvitz–Thompson full-test confusion-count estimates.Unweighted counts instead describe errors on the selected cases; precision and F1 are ratio estimates that may have finite-sample bias.Source: §1 · p. 2
  • The released sets contain 3,753 HI-Small and 2,268 LI-Small targets, reducing target counts by 271× and 611× while retaining full-graph neighborhoods for selected cases.The release includes GFP features and typed-subgraph prompts; the paper compares 12 budgets and seven alternative samplers and studies sampling error over 500 draws per configuration.Source: §1 · p. 2
  • Seven open-weight LLMs have weighted precision near full-test illicit prevalence, and lexical checks pass for over half of incorrect binary verdicts in the audit sample.A supplementary control attributes most of the apparent hybrid gain to the label-derived route.Source: §1 · p. 2

2 Related Work

Prior work explores LLM evaluation on balanced AMLworld subsets, graph reasoning from text, and efficient test-set estimation. AMLworld-Compact adds a full-test population reference while evaluating transactions from fixed graph neighborhoods.

LLM evaluation on AML graphs

  • Pirmorad (2025) explores in-context learning on a small balanced AMLworld subset; AMLworld-Compact reuses its typed-graph prompt template with different neighborhoods and demonstrations.Inclusion weights provide estimates for full-test detection despite the subset’s enriched illicit proportion.Source: §2 · p. 2

Graph inputs and graph retrieval

  • Chen et al. (2024) and Fatemi et al. (2024) study graph prediction and graph-text reasoning, whereas this work classifies transactions from fixed k-hop graph text.GraphRAG (Edge et al., 2024) retrieves over a graph index, a distinct problem from reasoning over a supplied neighborhood.Source: §2 · p. 2

Efficient evaluation and coreset selection

  • Unlike latent-ability benchmark methods such as tinyBenchmarks (Maia Polo et al., 2024) and metabench, this evaluation retains every illicit edge and allocates benign samples using supervised scores.CRAIG and CCS select training coresets; this work samples for evaluation and compares against stratified estimation and Neyman allocation.Source: §2 · p. 2

3 Constructing AMLworld-Compact

AMLworld-Compact selects test targets using HT-Coreset, then evaluates predictions with inclusion weights for full-test estimates. It retains all illicit cases, allocates benign samples across score-based strata, and supplies graph-text inputs and features for selected targets.

3.1 Setting

  • The study uses HI-Small and LI-Small synthetic transaction graphs, partitions source rows 60/20/20 in file order, and selects targets only from the test partition.The split is not strictly chronological because timestamps overlap across partitions; the released sets use a 1:2 illicit-to-benign ratio.Source: §3.1 · pp. 2–3

3.2 Class-Proportional Downsampling

  • Class-proportional sampling can require a 641K-edge HI-Small draw to meet the constrained 5 pp absolute F1-deviation criterion, while retaining all illicit cases permits a smaller benign budget.Importance weights account for unequal benign-edge selection probabilities.Source: §3.2 · p. 3

3.3 HT-Coreset

  • HT-Coreset retains all illicit edges and allocates the benign budget across a high-score stratum and three difficulty terciles, giving every benign edge positive inclusion probability.Selected edges receive inverse-inclusion weights for confusion-count estimates; recall is exact, while precision and F1 are ratio estimates with possible finite-sample bias and variance.Source: §3.3 · p. 3

3.4 Typed-Graph Serialization for LLMs

  • LLMs receive typed graph text extracted by breadth-first search from each target’s source account to depth two, considering incoming and outgoing neighbors.The evaluated prompts retain a literal <ID> placeholder and do not explicitly identify the target; graph-string token estimates exclude instructions, demonstrations, and outputs.Source: §3.4 · pp. 3–4, Table 1 · p. 4

3.5 Released Evaluation Set

  • The release aligns case IDs, labels, available typology annotations, GFP features, graph text, and HT weights, retaining all illicit edges and all eight typologies.Target selection uses seed 0 and is reproducible from the original construction scores.Source: §3.5 · p. 4

3.6 Metric Agreement and Held-Out Evaluation

  • For the construction ensemble, HT-Coreset has |∆F1|<10−13 pp across 12 budgets, with maximum tested reductions of 271× on HI-Small and 611× on LI-Small.Exact agreement follows from retaining that ensemble’s TP, FP, and FN contributors at weight one.Source: Figure 2 · p. 4
  • At released budgets, hard-negative sampling gives F1 RMSE of 0.00–2.58 pp versus 4.44–7.41 pp for uniform benign sampling, and has the lowest F1 RMSE in all 12 dataset–direction–budget comparisons.Table 2 evaluates held-out predictor families at released target counts; coverage is measured over 500 draws, and RMSE and mean 95% CI width concern F1 in percentage points.Source: §3.6 · p. 4, Table 2 · p. 5
  • The held-out estimates still vary when predictors flag benign edges outside the constructor’s hard stratum, and broad conservative intervals leave precise estimation for every predictor unestablished.Mean interval widths at released budgets are 27.3–55.0 pp.Source: §3.6 · p. 5

4 LLM Evaluation

The evaluation compares seven open-weight LLMs with GFP boosters under fixed typed-graph prompts and zero- or few-shot demonstrations. LLMs show high recall but little detection selectivity, while demonstrations improve typology scores more consistently than detection.

4.1 Evaluation Setup

  • The evaluation uses original typed-graph prompts, seven open-weight models, and zero-shot or few-shot variants with 12 demonstrations.GFP boosters receive features while LLMs receive graph text; ML thresholds were selected on full-test labels, making that comparison exploratory.Source: §4.1 · p. 5

4.2 Detection and Typology Results

  • 0.230%–0.253% HI-Small F1HT is near the 0.246% predict-all-illicit reference, with precision lift of 0.94–1.03.Zero-shot recall reaches 97.8%–100% for six models and 82.9% for Qwen3.5-27B; unweighted F1 is 40.7%–50.6%, below the ML reference’s 68.1%.Source: §4.2 · p. 6, Table 3 · p. 6
  • Across a ∼5.7× active-parameter range, F1HT varies by 0.017 pp, and deployed-model comparisons do not establish a controlled parameter-scaling effect.Qwen3.5-35B-A3B reaches 16.5% TF1, within 1.9 pp of Qwen3.5-397B’s highest score of 18.4%.Source: §4.2 · p. 6
  • Few-shot TF1 exceeds zero-shot TF1 for all seven models on HI-Small and six on LI-Small, with a mean HI-Small gain of 5.3 pp.Detection changes depend on the evaluation frame: mean unweighted F1 falls by 1.6 pp, while F1HT rises by 0.002 pp.Source: §4.2 · p. 6

5 Diagnosing LLM Outputs

The diagnosis examines ML–LLM error overlap, predicted labels and trace length, lexical checks, and a routing diagnostic. Across these analyses, written markers and high recall do not by themselves establish correct or selective detection.

5 Diagnosing LLM Outputs

  • The diagnosis uses records from run 42 for error and trace analyses, and written traces need not faithfully represent internal computation.The label-concentration figure instead pools all five runs.Source: §5 · p. 6

5.1 Decomposing ML and LLM Errors

  • Among 433 LLM-only-correct edges, 410 are illicit; among 2,123 ML-only-correct edges, 2,064 are benign.Both groups have mean ML scores far below the 0.80 threshold, so the ground-truth-defined groups do not supply an observable routing rule.Source: §5.1 · p. 6

5.2 Predicted Labels and Trace Length

  • False positives account for 38.7%–63.6% of HI-Small few-shot cases across models, while false negatives account for at most 4.7%.For ground-truth-benign cases, label concentrations vary by model and prompting; the figure pools percentages over five runs.Source: §5.2 · pp. 6–7, Figure 3 · p. 7
  • Few-shot traces are longer on average for every compared model, but correct and incorrect verdicts have no consistent mean-length ordering.These aggregate comparisons do not support using trace length as a confidence measure.Source: §5.2 · p. 7

5.3 Lexical Checks and Answer Correctness

  • 84.5% of binary-correct records and 56.2% of binary-incorrect records pass the revised lexical checks, so passing does not establish correctness.The outcome-stratified audit covers 1,000 HI-Small few-shot records; the checks detect written markers rather than verify graph support.Source: §5.3 · p. 8, Table 5 · p. 8
  • Human-rater Match agreement is low (κ = 0.10), and API judges receive no transaction graph, limiting validation of graph grounding and population failure rates.The human assessment covers a separate 28-record sample.Source: §5.3 · p. 8

5.4 A Supplementary Routing Diagnostic

  • Doubt Triage reaches 87.3% HI-Small and 84.9% LI-Small zero-shot F1, but an always-illicit answer on the same route reaches the same scores.The label-derived route accounts for most of the apparent gain and does not establish a deployment selection rule.Source: §5.4 · p. 8

6 Discussion and Conclusion

AMLworld-Compact supports model and prompt comparisons with thousands of targets while retaining every illicit test case. Its sampling study favors the tested benign allocation over uniform sampling, but conservative intervals remain broad; the evaluated LLMs show limited detection selectivity and lexical checks also pass on many incorrect outputs.

6 Discussion and Conclusion

  • The released evaluation sets enable comparisons with thousands of targets while retaining every illicit test case.Align predictions to the released case IDs and apply inclusion weights to confusion counts for full-test estimates; report unweighted errors for selected cases.Source: §6 · p. 8
  • Holding out a GFP booster family gives lower point error than uniform benign sampling at matched budgets, but conservative intervals remain broad.This supports the allocation for tested families while leaving transfer to other predictors open.Source: §6 · p. 8
  • Under the original graph prompts, evaluated LLMs flag many benign transactions, and lexical checks pass for over half of incorrect binary verdicts.The discussion recommends reporting typology and detection separately and comparing lexical checks with answer correctness.Source: §6 · p. 8

Limitations

The evaluation is limited to small synthetic AMLworld splits and one graph-to-text pipeline, with target identification inconsistent in the evaluated inputs. Sampling and trace conclusions also have bounded validation: several checks use frozen designs or single-run evidence.

Limitations

  • The seven-model evaluation covers two small synthetic splits and one graph-to-text pipeline; generalization to larger graphs and real banking data remains untested.Inputs do not consistently identify and include the target transaction, so errors cannot be attributed solely to model reasoning.Source: Limitations · p. 8
  • Overlapping timestamps in CSV-order partitions do not establish strictly prospective ML performance.Thresholds for the task ML reference and DT were selected on full-test labels, making those comparisons exploratory.Source: Limitations · p. 8
  • Sampling uncertainty is conditional on a frozen design and predictions, without validation against full-test LLM predictions or independent coreset draws.The held-out-family study tests two related GFP boosters, not transfer to unrelated predictors.Source: Limitations · p. 9
  • Trace-level findings come from the available run, and their stability across runs is untested.The 28-record human check assesses rubric application on an outcome-stratified sample.Source: Limitations · p. 9

Ethical Considerations

AMLworld is synthetic, and the paper identifies potential harms from both laundering-pattern examples and false positives. The authors state that the reported over-prediction does not support LLM-only deployment.

Ethical Considerations

  • AMLworld’s account identifiers, amounts, and relationships do not expose real customers.The paper also notes that laundering-pattern examples could aid attempts to evade detection, while false positives can burden legitimate customers and compliance teams.Source: Ethical Considerations · p. 9
  • Reported over-prediction does not support LLM-only deployment.Retaining ML in the loop does not establish deployment safety.Source: Ethical Considerations · p. 9
  • The evaluation is intended to improve AML detection, but examples of laundering patterns could also help attempts to evade detection.Release licenses and the code-only Elliptic distribution are documented in Appendix A.Source: Ethical Considerations · p. 9

A Dataset Details

This appendix details the dataset, graph prompts, coreset construction, estimation scope, and sampling studies. Together, these choices preserve illicit cases while reducing evaluation targets, but evidence about estimator performance is conditional on the evaluated predictors and designs.

B Graph Inputs and Target Identification

  • The original prompt extraction uses k=2 and caps expansion at 50 neighboring accounts per expanded account per hop, while retaining transaction edges among visited accounts.The fixed depth and cap were not ablated; few-shot demonstrations use k=1 and at most 15 neighbors per hop.Source: B Graph Inputs and Target Identification · p. 11

B.1 Target Identification in the Original Prompts

  • Original prompts omit the target transaction in 324 HI-Small and 297 LI-Small cases, and even present targets lack an explicit target marker.False-positive flag rates among target-present benign cases remain high, but group differences do not isolate an effect of target omission.Source: §B.1 · p. 11

B.2 Target-Marked Inputs and Integrity Checks

  • The targeted-input release identifies and marks each target, inserting absent targets while removing one nontarget edge to preserve edge-count budgets.The combined input changes have not been evaluated for prediction effects or compatibility with model context limits.Source: §B.2 · p. 12

C Model Settings and Compute Budget

  • A zero-shot compact-set pass uses 98–165M tokens on HI-Small and 51–102M on LI-Small.The archived GCPAL inputs combine 79 GFP and raw-attribute dimensions with five line-graph summaries; total LLM inference compute is approximately 2,840 GPU-hours.Source: C Model Settings and Compute Budget · p. 12

D.2 Estimation Properties and Scope

  • For fixed predictions and known positive inclusion probabilities, HT weights give unbiased full-test confusion-count estimates; retaining every illicit edge makes recall exact.Precision and F1 are ratio estimates that may have finite-sample bias and sampling variance; the conservative intervals concern fixed prediction vectors under the frozen design.Source: §D.2 · pp. 13–14

D.3 Naïve Coreset vs HT-Coreset

  • At released sizes, Naïve Coreset reduces targets only 1.6× on HI-Small and 1.2× on LI-Small, retaining 790/1,251 illicit edges in its HI-Small draw.HT-Coreset retains every illicit edge, avoiding this source of recall variation.Source: §D.3 · p. 14

D.4 Construction-Ensemble Comparison

  • HT-Coreset has the smallest deviations among seven alternative samplers in the construction-ensemble comparison.Other methods either captured few illicit edges, deviated in precision, or ignored class labels; the comparison uses 50 draws with the original ensemble.Source: §D.4 · p. 15, Table 9 · p. 15

D.5 Sampling with a Held-Out Predictor Family

  • The held-out-family study evaluates one GFP booster family using strata built by the other, with 500 draws at benign budgets of 2, 4, and 8 times the illicit count.Its confidence intervals concern each fixed five-seed configuration, and the shared GFP features do not establish transfer to arbitrary predictors.Source: §D.5 · p. 15

D.6 Supervised Baselines and Data Splits

This block reports supervised references, per-model sampling agreement, and an Elliptic external check. It also specifies the scoring convention for typology results; the supplied passages do not include the full LLM detection results.

D.6 Supervised Baselines and Data Splits

  • The two-booster reference has pooled full-test F1 of 68.1% on HI-Small and 28.6% on LI-Small.The table reports full-test detection metrics as mean±std across five seeds; individual boosters use validation-selected thresholds.Source: §D.6 · p. 16, Table 11 · p. 16

D.7 Per-Model Sampling Comparison

  • HT-Coreset’s per-model F1 deviations are small for LightGBM and XGBoost, but 3.9 pp on HI-Small and 2.6 pp on LI-Small for GCPAL.The exact construction-ensemble agreement does not extend to every constituent scorer; all compared models use GFP features.Source: §D.7 · p. 17, Table 12 · p. 17

D.8 Sampling Evaluation on Elliptic

  • On Elliptic, HT-Coreset retains all 1,083 illicit nodes plus 2,166 benign nodes and reduces the labeled test set 5.1×, with exact P/R and |∆F1|<10−13 pp.This is for the evaluated ensemble and construction; B3 also matches here because its false positives lie in the fully retained hard-negative stratum.Source: §D.8 · p. 18, Table 13 · p. 18
  • Elliptic’s exact agreement depends on retaining all nodes contributing to the evaluated ensemble’s confusion counts, and its B6/B7 adaptations differ from AMLworld’s.The B6 comparator lacks known inverse inclusion probabilities, so the HT guarantee does not apply to it.Source: §D.8 · p. 18

E Full LLM Results

  • Typology F1 uses only illicit edges with known reference typologies and averages class F1 over the union of reference and predicted labels, including none or legitimate when present.The released live parser does not fully reproduce every historical free-text fallback, so exact table reproduction uses archived labels.Source: E Full LLM Results · p. 18

E.1 Full Results and Between-Run Variation

Weighting changes the HI-Small compact-F1 ordering, while uncertainty intervals qualify model comparisons and few-shot gains. Error analyses distinguish detection from typology classification and show that Doubt Triage uses label-informed routing.

E.1 Full Results and Between-Run Variation

  • 50.6% compact F1 for Qwen3.5-397B ICL-FS contrasts with the largest HI-Small HT estimate, 0.2525%, for Qwen3.5-35B-A3B ICL-FS.Compact and HT ordering correlations are 0.64 on HI-Small and 0.94 on LI-Small; these descriptive orderings do not establish population model ranks.Source: §E.1 · p. 18

E.2 Sampling Uncertainty

  • All seven HI-Small FS–ZS intervals include zero; on LI-Small, only GPT-OSS-20B has an interval excluding zero, with ∆F1 = −0.0111 pp.The intervals do not establish prompt equivalence, and coverage is per comparison.Source: §E.2 · p. 19, Table 16 · p. 21

E.3 Few-Shot versus Zero-Shot

  • Few-shot TF1 exceeds zero-shot TF1 for all seven HI-Small models and six LI-Small models, while compact-set detection-F1 changes remain small and mixed.Compact-set detection-F1 changes range from −5.0 to +0.5 pp on HI-Small and −4.5 to +0.5 pp on LI-Small; Table 16 separately reports HT-weighted differences with uncertainty.Source: §E.3 · p. 20, Table 16 · p. 21

F.1 ML Scores by Correctness Group

  • LLM-only-correct edges are mostly illicit, whereas ML-only-correct edges are mostly benign; their low mean ML scores provide no observable routing rule.These groups are defined by ground-truth correctness.Source: §F.1 · p. 21

F.3 Over-Prediction Ratio and Per-Typology Recall

  • The run-42 over-prediction ratio spans 1.71–3.00 on HI-Small and 2.11–3.00 on LI-Small across models and prompts.For six zero-shot models, excluding Qwen3.5-27B, it is 2.94–3.00 and 2.91–3.00, respectively; 3 is the predict-all value for the 1:2 subset.Source: §F.3 · p. 21

F.2 Typology Classification after Correct Detection

  • 72.7% ensemble macro-F1 versus 6.6–15.8% for LLMs is conditional on correctly detected illicit edges.The comparison uses true positives, unlike Table 3’s TF1.Source: §F.2 · p. 21, Figure 4 · p. 22

F.5 Typology Concentrations by Model and Prompting

  • Among predicted-illicit typologies, the dominant label’s share ranges from 35% to 91% on HI-Small and 42% to 91% on LI-Small across models and prompts.For GPT-OSS-120B on HI-Small, fan-out accounts for 52% under ICL-FS and gather-scatter for 70% under ICL-ZS.Source: §F.5 · p. 23

G.1.1 Routing Rule

  • Doubt Triage routes LLM consultation using HT-Coreset strata that incorporate ground-truth labels, making it an optimistic diagnostic rather than a deployable policy or mathematical upper bound.It consults the LLM on ML-negative edges in the weight-one stratum and uses ML elsewhere.Source: §G.1 · p. 23, §G.1.1 · p. 23

G.1.2 Results

Doubt Triage exceeds the ML reference in tested cells, but same-route controls and label-derived strata limit what that comparison establishes. Score-only gates and trace audits likewise have bounded scopes, including lexical checks that can pass incorrect answers.

G.1.2 Results

  • Doubt Triage exceeds ML in all 140 tested cells, but exceeds the same-route always-illicit control in only 14/70 cells per split.The comparison does not isolate LLM value beyond the label-derived route or establish generalization across datasets.Source: §G.1.2 · p. 23

G.2 Score-Only and Naïve Hybrid Rules

  • The best tested score-only gate cells reach 66.4% F1 on HI-Small and 37.2% on LI-Small, but thresholds were selected on evaluation predictions.The search omits some narrower deferral bands, so its below-ML mean does not rule out better score-only gates.Source: §G.2 · p. 24

H Reasoning-Trace Analysis

  • The 1,000-trace audit is outcome-stratified, and its group counts are sampling outcomes rather than population shares.The slice contains 183 correct-illicit, 9 correct-benign, 295 over-prediction, 235 under-prediction, and 278 typology-error records.Source: H Reasoning-Trace Analysis · p. 24

H.1 Automated Audit by Model and Annotator

  • Figure 5 breaks model-level regex pass rates out from pooled audit rates; the audit includes 148 empty traces and API judges’ text windows truncate 400 traces.The table reports pooled rates and agreement across four annotators.Source: §H.1 · p. 24, Figure 5 · p. 25, Table 22 · p. 25

H.2 Sensitivity to Scoring Rules and Text Windows

  • After removing the benign-verdict bypass, 298 binary-error records still pass lexical checks, or 295 in the shorter window.These are sampled-record comparisons, not population prevalence or a causal failure stage.Source: §H.2 · p. 25
  • Of 760 records evaluable for binary and available-typology correctness, 417 pass revised lexical checks but fail answer correctness, or 412 in the shorter window.The 240 illicit records without a reference typology are excluded.Source: §H.2 · p. 25

H.3 Human Ratings of the Step Rubric

  • Human Match agreement is low (κ = 0.10), while six of seven correct-illicit traces pass Conclude under the shared parser rule.The human ratings are from a separate 28-trace sample and do not estimate population rates or paired human–judge disagreement.Source: §H.3 · p. 26

H.4 Trace-Length Statistics

  • Table 25 combines trace availability and length across prompting methods with few-shot trace lengths conditioned on binary correctness.The section points readers to these statistics for interpretation but provides no values in the supplied passage.Source: §H.4 · p. 26

H.5 Prediction and Deployment Error Examples

The examples include context-budget rejections and a false positive whose AML vocabulary does not establish correctness. Further prompt probes show variant-specific outcomes, but their comparisons are limited to a small, differently structured evaluation pool.

H.5 Prediction and Deployment Error Examples

  • Two models assign different typologies to the same suspicious graph: GPT-OSS-120B selects cycle, while Qwen3.5-397B selects scatter-gather.Both traces identify the hub and repeated transfers but emphasize different structures.Source: §H.5 · p. 26
  • A GPT-OSS-120B false positive uses graph features and laundering-pattern language, but the example does not establish which internal computation caused the error.The case is a 526-transaction benign subgraph.Source: §H.5 · p. 26
  • 1,443 of Qwen3.5-27B’s 1,453 empty outputs carry the same context-length error, explaining most of its 38.7% no-response rate.The model’s native 256K context was deployed at 128K because of memory constraints under four-way tensor parallelism.Source: §H.5 · p. 27

I.1 Main Evaluation Prompts

  • The main few-shot prompt follows Pirmorad (2025) in format and uses an 8+4 demonstration structure; its random-typology example describes sub-threshold structuring, not AMLworld’s random-walk pattern.The same prompt was retained for all reported results.Source: §I.1 · p. 27

I.2 Verification Prompt

  • The verification prompt adds a structural-marker requirement and legitimate-pattern checks; across all four model-dataset cells, precision rises and recall falls, while F1 changes have mixed signs.These are unweighted compact-set metrics from run 42.Source: §I.2 · p. 27, Table 26 · p. 27

I.3 Frontier API Probe: Typology Prediction

  • The frontier probe compares six prompt variants on three APIs using 198 cases, with 22 cases per class across eight illicit typologies and benign.Its instructions, graph inputs, class mix, and scoring differ from the main evaluation, so comparisons concern variants within this probe.Source: §I.3 · pp. 27–28
  • FS-TypFirst has the highest Typ-F1 among tested variants for each API; ZS-Abstain lowers detection recall relative to ZS-Graph and reaches at least 90% precision.The highest typology macro-F1 is 30.3% for DeepSeek-V4-Pro FS-TypFirst; the probe does not isolate pre-training, model size, or thinking-budget effects.Source: §I.3 · p. 28

I.3.1 Prompt Specifications

  • The probe’s FS-TypFirst prompt uses eight suspicious examples and no benign examples for an upstream-flagged case, deriving its verdict from the stated pattern.It returns Explanation, Observed Pattern, then Conclusion.Source: §I.3.1 · p. 28
Loading 2610.13627v1…