Source-linked AI summary
PaMIR: Open Benchmark of Public Credit-Default Datasets
Mikhail Liashkov, Ilyas Varshavskiy, Shuhratjon Khalilbekov, Azizjon Azimi, Bonu Boboeva
TL;DR
Credit-default research has a narrow empirical base, and deployed models face labels that arrive only after maturation. PaMIR assembles public datasets and evaluates models with repeated i.i.d. splits and arrival-ordered, label-delayed streams; its collection includes 19 datasets, more than twice the size of each reference benchmark study.
Problem
Credit-scoring comparisons use a narrow empirical base, and ordinary time-based splits can include outcomes that would not yet have matured when the test period begins.
Source: §1 · p. 3Method
PaMIR rebuilds public credit-default datasets with a leakage-audited recipe and evaluates single-function models on repeated i.i.d. splits and arrival-ordered, label-delayed streams.
Source: §1 · p. 3Results
19 public credit-default datasets form PaMIR's collection, more than twice the eight datasets in each reference benchmark study, of which only two and four are public.
Source: §1 · p. 3Takeaways & Limitations
The one-function interface lets the same 19 tables serve as a credit track for tabular foundation models and support study of whether they change credit-risk prediction.
Source: §1 · p. 3The collection mixes datasets of unequal informativeness, while its unweighted fleet mean treats all 19 as equally informative.
Source: §8 · p. 23
Abstract
from arXiv · showhide
We release PaMIR (Public Arrival-ordered Measurement for Inference in Risk), an open benchmark for credit-default prediction when labels are scarce and arrive late. The field's reference benchmark studies use eight datasets each, only two or four of them public. PaMIR brings together 19 public datasets with binary default labels -- 1.24M loans, firms and card accounts from nine countries -- rebuilt from pinned source snapshots by one leakage-audited recipe and never redistributed; to our knowledge it is the one of its kind as of today. Every model is a single function, scored under a repeated i.i.d. split and a label-delayed stream in which each application is scored on arrival, with AUC reported by label budget; fleet means are withheld unless every dataset is scored. A synthetic-data harness tests generated training rows without letting a generator see held-out rows. This report describes release 0.4.0 of this living benchmark.
1 Introduction
PaMIR addresses credit-default prediction with scarce, delayed labels and a narrow, leakage-prone empirical base. It contributes a public dataset collection, evaluation protocols for delayed labels, and a leakage-controlled synthetic-data harness.
1 Introduction
- Credit-scoring comparisons rely on a narrow empirical base, while post-outcome columns can inflate performance and ordinary time splits may include labels not yet matured.Reviews report 2.75 datasets on average for papers published from 2016 to 2020; reference studies use eight datasets each, only two and four of them public.Source: §1 · p. 3
- PaMIR is a living benchmark with versioned dataset additions and is released as an open-source package with a recipe-based path for adding datasets.This report describes release 0.4.0.Source: §1 · pp. 3–4
- 19 public credit-default datasets, rebuilt from pinned snapshots under a leakage-audited recipe, form an open collection whose data are fetched from origin and never redistributed.The collection covers consumer, peer-to-peer, credit-card, vehicle, small-business and corporate lending.Source: §1 · p. 3
- Arrival-scored, label-delayed streams report AUC by label budget, while repeated i.i.d. splits enable comparison with general benchmarks.Fleet summaries are withheld for partial runs, failed data contracts, or scores that depend on other rows.Source: §1 · p. 3
- A synthetic-data harness fits generators inside each fold, reports paired utility deltas and fidelity, and probes for reproduced held-out rows.Source: §1 · p. 3
2 Related work
PaMIR places its public credit-default collection against credit-specific and general tabular benchmarks, and remains larger than the largest prior public collection under a stricter source count. Its label-delayed stream and synthetic-data protocol address evaluation settings that the related benchmarks do not cover in the same way.
2.1 Multi-dataset credit studies, and why PaMIR is the largest
- The comparison counts datasets with binary default, bankruptcy, or financial-distress targets used together, separates public from proprietary data, and excludes loss-given-default and non-default tasks.The literature search covered reviews, reference benchmarks, general tabular benchmarks, and recent multi-dataset credit studies.Source: §2.1 · p. 4
2.2 Other benchmarks and related work
- OpenML-CC18 curated 72 classification datasets, TabArena [13] has 51, MultiTab [14] organizes 196, and BeyondArena [15] has 142.PaMIR is narrower in domain than these general benchmarks but broader within credit evaluation.Source: §2.2 · pp. 4–6
2.1 Multi-dataset credit studies, and why PaMIR is the largest
- 19 public datasets exceed the prior study counts, and 14 remain after strict consolidation versus 12 public datasets in the largest prior study.The strict count merges the Polish horizons and LendingClub-derived tables and drops the synthetic dataset.Source: §2.1 · p. 4, Figure 2 · p. 5
2.2 Other benchmarks and related work
- Calendar-date splits measure robustness to calendar drift, whereas PaMIR makes loan-maturation delay explicit, so the two evaluation types complement each other.The cited temporal credit splits do not leave a gap between training and test periods.Source: §2.2 · p. 6
- PaMIR applies delayed-label stream evaluation to credit portfolios, treating loan maturation as the delay and scheduling refits by resolved defaults rather than wall time.Prior work studies stream evaluation and delayed labels, while PaMIR applies that structure to credit portfolios.Source: §2.2 · p. 6
- PaMIR’s synthetic module fixes an evaluation protocol for claimed utility gains rather than proposing a generator.Existing synthetic-data evaluations often use splits the generator has seen.Source: §2.2 · p. 6
3 The PaMIR collection
PaMIR assembles 19 public credit-default datasets under shared inclusion, rebuilding, and leakage-audit rules. It pins sources and validates rebuilt tables against frozen contracts, while admitting additions only through a versioned recipe process.
3.1 Inclusion criteria
- Datasets must be single-row-per-obligor, public, binary-default tables with at least 1,000 rows and a default rate of at least 3%.Sources may be reduced to the required table by a documented join or aggregation.Source: §3.1 · p. 6
3.2 Composition
- 19 datasets contain 1,237,550 rows and 281,766 defaults across five product families, with sizes from 1,000 to 266,482 rows and features from 10 to 94.Nine countries are named, and default rates range from 3.2% to 40.9%.Source: §3.2 · p. 7, Table 3 · p. 7, Figure 3 · p. 8
3.3 Provenance, retrieval and validation
- All 19 tables were rebuilt from pinned snapshots and met frozen contracts recording table structure, counts, and raw-file and target digests.A mismatch raises an error and prevents caching; the check is repeated when cached tables are loaded.Source: §3.3 · pp. 7–8
3.4 Harmonization recipe
- A six-step recipe standardizes targets and features, removes identifiers and columns unavailable at decision time, then shuffles rows with a fixed seed.The semantic day-zero cut replaces a univariate-AUC threshold that can miss jointly predictive post-outcome columns; SBA term divisibility is one documented leak.Source: §3.4 · pp. 8, 10, Figure 5 · p. 10
3.5 Leakage audit
- The audit rejected seven candidates for recoverable labels, rule-generated labels, prohibited use, or independently shuffled columns, rather than patching them.Each guard has positive and negative controls; data-dependent checks run in continuous integration on datasets outside Kaggle and wherever all 19 are cached.Source: §3.5 · p. 11, Figure 6 · p. 11
3.6 Licensing and redistribution
- PaMIR distributes code and recipes, not datasets; users fetch each table from its source and remain subject to that source’s terms.The catalogue records declared licences and obligations; some Kaggle re-upload licences do not transfer rights in the underlying data.Source: §3.6 · p. 12
3.7 A living collection
- New datasets enter as locally rebuilt, audited recipes whose validated results are frozen in expected.json and shipped through versioned releases.For Kaggle-hosted additions, contributors run the data checks locally because continuous integration downloads only the seven datasets outside Kaggle.Source: §3.7 · pp. 12–13, Figure 8 · p. 13
4 Evaluation protocols
PaMIR evaluates one model interface under repeated i.i.d. splits and a stream where scores are fixed when applications arrive, using only matured labels. Its reporting rules expose label-budget performance and require complete, valid coverage for a fleet mean.
4.1 Model interface and metric
- The shared model interface uses fit_fn to return a scorer, while the i.i.d. interface accepts predict_fn; the metric is ROC AUC with Gini also reported.Scoring rows must be independent: the streaming evaluator re-scores a random subset of each batch to test this condition.Source: §4.1 · pp. 13–14, §4.3 · p. 14
4.2 i.i.d. protocol
- The i.i.d. protocol permutes each dataset, trains on 70% of rows, scores the remainder once, and averages AUC across five seeds with standard deviation.By default it uses each full table.Source: §4.2 · p. 14
4.3 Streaming protocol
- The streaming protocol scores each application once on arrival using a model fitted only to labels matured by then, and reports AUC by resolved-label budget.Refits follow resolved-default arrivals; pre-first-refit rows are unscored, while the last delay-length rows are scored and included in AUC after their labels resolve.Source: §4.3 · p. 14, Figure 9 · p. 16
- Because rows are randomly permuted, the stream is exchangeable and measures cold-start ranking with delayed, schedule-limited labels, not calendar drift.The effective delay is capped at 20% of stream length; release 0.3.0 refresh-mode results are not comparable with arrival-mode results.Source: §4.3 · p. 14
4.4 Reference setting and reporting
- 0.7213, 0.7174, and 0.7132 are the arrival-mode AUCs for logistic regression on taiwan at k = 10, 40, and 100, respectively.The corresponding runs made 105, 27, and 11 refits; comparisons require matching mode, lag, refit cadence, and maximum row count.Source: §4.4 · p. 15
4.5 Coverage is part of the result
- A fleet mean is reported only for complete runs whose tables pass contract checks and whose scores pass the row-independence probe.On six datasets, full-coverage logistic regression averaged 0.7177 AUC, while a variant failing on four datasets reported 0.7294 over the two it scored.Source: §4.5 · pp. 15, 17, Figure 9 · p. 16
5 Leakage-controlled synthetic augmentation
The synthetic-augmentation harness measures whether generated training rows improve discrimination on held-out real rows, while controlling leakage and assessing fidelity. Its instrument check shows that different fidelity measures can expose defects hidden by an aggregate score.
5.1 Estimand and composition
- The estimand is the paired AUC difference between augmented and real-only training under the same protocol, interpretable only when generator fitting and AUC evaluation use disjoint rows.Default sizing keeps all real training rows and adds synthetic rows; fixed-total sizing instead pins the frame size and subsamples real rows.Source: §5.1 · p. 17
5.2 The leakage contract
- The leakage contract fits generators only on training rows, resets them for each fold, keeps held-out rows out of augmentation, and probes for reproduced held-out rows.Precomputed generators or pools must be fold-bound or explicitly vouched for; row normalization makes the reproduction probe robust to type differences.Source: §5.2 · p. 17
5.3 Fidelity battery and protocol
- The fidelity battery combines library reports, generator-provided metrics, and a lightweight set of distributional, correlation, two-sample, and nearest-record measures.The classifier two-sample AUC is interpreted by distance from 0.5; the full report yields 66 fidelity columns per fold and generator.Source: §5.3 · pp. 17–18
- The default evaluation uses five-fold stratified cross-validation, a paired real-only baseline, pooled out-of-fold AUC, and fold-wise delta summaries.Generators are fitted within each fold, and the held-out rows are scored but never augmented.Source: §5.3 · p. 18, Figure 11 · p. 19
5.4 An instrument check
- 0.7669 to 0.7438 (∆ = −0.0231): the synthetic mix lowered pooled out-of-fold AUC on south_german, with fold-wise mean −0.0273 and only two of five folds positive.The fold-wise standard deviation was 0.0619, more than twice the effect; leakage probes were clean in all ten generator–fold pairs.Source: §5.4 · p. 18
- SDV’s aggregate score ranked marginal above joint (0.953 against 0.848), despite marginal’s lower correlation similarity (0.888 against 0.988); the two-sample test distinguished joint perfectly (AUC 1.000).Table 5 reports fidelity averaged over five folds; the closest-record distance also separated the generators’ failure modes.Source: §5.4 · p. 18, Table 5 · p. 20
6 Software and reproducibility
PaMIR packages its benchmark as a Python tool with command-line dataset management and documented evaluation workflows. Its tests cover data integrity, protocols, baselines, and the synthetic module, while reported runtimes quantify the cost of reference and synthetic runs.
Package and interface
- PaMIR is a pure-Python package with command-line commands to list, inspect, download, and locate cached datasets.The core needs no GPU; data-fetching and synthetic-data dependencies are optional extras.Source: §6 · p. 20
Testing and documentation
- The 206-test suite covers catalogue and recipe integrity, data contracts, leakage guards, both streaming modes, reference baselines, data quality, and synthetic evaluation.Tests requiring data skip cleanly when a dataset is not cached; documentation is also exported as llms.txt and llms-full.txt.Source: §6 · p. 20
Cost
- A full-fleet reference run took about 5 minutes for logistic regression and 16 minutes for GBDT across 19 datasets on the stated CPU; full-table i.i.d. runs took under a minute per baseline.A five-fold synthetic run with two generators took about two minutes, mostly for measurement rather than generation.Source: §6 · p. 20
7 Reference baselines and results
Across the reference runs, GBDT generally outperformed logistic regression under both i.i.d. and arrival-mode evaluation, but its advantage was smaller in the stream. Streaming scores were lower than i.i.d. scores, and the label-budget results show that the relative performance changes as more labels mature.
Protocol gap
- The stream scored lower than i.i.d. on all 19 datasets for GBDT, by 0.032 in fleet mean, and on 16 of 19 for logistic regression, by 0.009.The report notes that most stream rows are scored by models trained on fewer labels than the i.i.d. split’s 70% training share.Source: §7 · p. 21
Ranking
- The better baseline changed between i.i.d. and stream evaluation on three datasets; GBDT led under i.i.d. and trailed under the stream on each.Source: §7 · p. 21
Delay and refresh
- Shortening the delay from 1,000 to 250 positions raised fleet mean AUC by 0.002 for both baselines; the 0.3.0 refresh rule scored higher than arrival mode by 0.001 for logistic regression and 0.003 for GBDT.For the shorter delay, the paired dataset counts were 11 for logistic regression and 13 for GBDT; for refresh, they were 12 and 16, respectively.Source: §7 · p. 21
Headroom
- 0.060: under the i.i.d. split, GBDT exceeded logistic regression in fleet mean AUC (0.785 against 0.725) and on 14 of 19 datasets; its stream margin was 0.036 (0.753 against 0.717) on 11 of 19.The paired Wilcoxon p-values were 0.004 for i.i.d. and 0.20 for arrival mode.Source: §7 · p. 21, Table 6 · p. 22
Label budget
- Below 300 resolved labels, fleet mean AUC favored logistic regression; from 300 labels onward, GBDT led, by 0.039 and 0.053 in the groups between 1,000 and 9,999 labels.The rows in each label group were the same for both models because they shared a refit schedule.Source: §7 · p. 21, Table 7 · p. 22
Reference plan
- The full-table i.i.d. fleet mean rose from 0.725 to 0.726 for logistic regression and from 0.785 to 0.794 for GBDT, compared with the 20,000-row truncation.The reference plan reported both baseline protocols across all 19 datasets, with fleet means defined because every run passed its validity checks.Source: §7 · pp. 21–22
8 Limitations
PaMIR’s limitations concern the temporal scope of its replay, the limits of batch-independence checks, heterogeneous source quality, synthetic-module coverage, and the narrow baseline set in release 0.4.0.
8 Limitations
- The stream uses a fixed random permutation, measures delay in stream positions rather than months, and does not model population drift.The delay cap also shortens delays on the four smallest datasets.Source: §8 · pp. 22–23
- Batch scoring can conceal cross-row dependence below the probe’s numerical tolerance or outside the re-scored subset.Setting batch_scoring=False avoids this uncertainty but requires one scorer call per row.Source: §8 · p. 23
- The collection mixes data of varied quality and provenance, and its unweighted fleet mean treats all 19 datasets as equally informative.Some sources may change or disappear; a withdrawn source fails its contract rather than being replaced.Source: §8 · p. 23
- The synthetic harness uses cross-validation rather than streaming, and its leakage probe detects exact reproduction but not slightly perturbed held-out rows.Pair metrics are capped, and closest-record distance uses at most 2,000 rows per side, understating memorization on larger tables.Source: §8 · p. 23
- Release 0.4.0 reports two untuned package baselines; tuned learners and tabular foundation models are not evaluated.At the reference setting, arrival-mode evaluation would require 773 refits or re-conditioning on bondora and calls for a GPU.Source: §8 · p. 23
9 Roadmap
PaMIR is planned as a maintained benchmark, with future releases expanding datasets, protocols, reference results, and infrastructure. Its authors expect the protocol machinery to transfer to other tabular prediction settings that share the identified challenges.
9 Roadmap
- Future releases aim to expand the collection with additional public probability-of-default datasets admitted through the existing audit.The stated aim is to pass 20 datasets.Source: §9 · p. 23
- Temporal tracks would use origination dates and out-of-time windows, with a training–test gap matching the maturation period.These tracks need a censoring correction because terminal-status-only samples over-represent early defaults in recent cohorts.Source: §9 · p. 23
- Planned protocols include controlled shifts of graded severity, low-default tracks, and tail-focused metrics.The paper notes that rare events carry costs in credit books and Gaussian-prior generators can struggle with heavy tails.Source: §9 · p. 23
- The synthetic harness is planned for extension to streaming and shifted settings, where synthetic data are often claimed to help.The current harness does not yet run under streaming, as noted among the limitations.Source: §9 · p. 23
- The roadmap adds reference results for deep tabular and foundation models, plus generators including copulas, CTGAN, TabSyn, and TabDiff [31] [32] [33].Source: §9 · p. 23
- Planned infrastructure includes DataFoundry export, a frozen v1.0 specification, datasheets, verified submissions, and governance guidance.The authors expect PaMIR’s protocol machinery to transfer because rare positives, delayed outcomes, heterogeneous leaky sources, and synthetic-data evaluation concerns recur across tabular prediction.Source: §9 · pp. 23–24
10 Availability
The report describes release 0.4.0 and provides code, recipes, and documentation under Apache-2.0. PaMIR is versioned, and users are asked to cite the report and each dataset’s original source.
10 Availability
- Code, recipes, and documentation are released under the Apache-2.0 license.The package is named pamir-credit.Source: §10 · p. 24
- The documentation and Hugging Face dataset-card links are identified as provisional or forthcoming.Source: §10 · p. 24
- PaMIR is updated through versioned releases; users should cite this report and the original source of every dataset they use.This report describes release 0.4.0.Source: §10 · p. 24
Conflict of interest
All authors are affiliated with zypl.ai, which develops synthetic-data generators for credit risk. The report distinguishes the company’s generators from the benchmark’s evaluated reference results.
Conflict of interest
- All authors are affiliated with zypl.ai, which also develops credit-risk synthetic-data generators including zGAN and zEDGE.Source: Conflict of interest · p. 24
- The package includes adapters for zGAN and zEDGE, but the generators are distributed separately and are not evaluated in the report or reference results.Source: Conflict of interest · p. 24
- Any future evaluation of a maintainer-developed model or generator will use the published protocol, include its code, and be marked as such.Source: Conflict of interest · p. 24
A Dataset provenance
Table 8 documents each dataset’s download source and default-label definition; datasets are fetched and rebuilt locally from named sources.
A Dataset provenance
- Each dataset is fetched from a named source and rebuilt locally.Table 8 records each source and the dataset’s target definition.Source: Table 8 · p. 27, A Dataset provenance · p. 27
- Pinned snapshots and target definitions document each dataset’s provenance.UCI archives have no version and are pinned by digest alone.Source: Table 8 · p. 27
- Raw files are pinned by SHA-256 digest, and full recipes include day-zero drop lists.Recipes are available through pamir.dataset_info(id).Source: Table 8 · p. 27