PaperlayerView exact revision on arXiv

Source-linked AI summary

A Decision-Support Audit Protocol for Supervision Drift in Proxy-Labeled Credit-Risk Prediction

Mehrdad Shoeibi, Muhammad Shabanpour, Waldemar Karwowski, Niloofar Yousefi

arXiv:2609.16102v1cs.LGcs.AI

TL;DR

Credit-risk models use proxy labels and face changing periods and market segments, but a single transfer metric cannot distinguish prevalence, probability-scale, and relationship changes. The paper introduces a locked multi-signal audit protocol and demonstrates it on LendingClub transfer settings. Ranking was stable and oracle gaps small, while the clearest temporal signal was a prevalence and probability-scale mismatch; its cause remains unidentified from the release.

  • Problem

    A single transfer metric cannot distinguish base-rate shift, probability-scale shift, and feature-label relationship change in proxy-labeled credit-risk prediction.

    Source: §1 · pp. 1–2

  • Method

    The paper demonstrates a locked five-layer audit protocol with pre-specified thresholds and decision rules for proxy-labeled credit-risk supervision drift.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Ranking transfer was stable and oracle gaps small; the clearest temporal signal was a prevalence and probability-scale mismatch that diagnostic recalibration largely reduced.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The protocol characterizes bounded shift evidence for decision support but does not attribute the observed mismatch to a specific mechanism.

    Source: §1 · p. 2, §3.1 · p. 3, §6 · p. 10

    The study uses one LendingClub dataset, and the release cannot separate population change from maturity-related selection or other cohort-composition effects.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

Credit-risk models are trained on proxy labels and deployed under temporal and segment change, yet no single transfer metric separates base-rate shift, probability-scale shift, and feature-label relationship change. We contribute a design-science artifact: a locked, multi-signal audit protocol for supervision drift in proxy-labeled credit-risk prediction. Five layers (transfer performance, an oracle-gap probe, a calibration diagnostic, feature-label stability, and a synthetic positive control), thresholds, and decision rules were locked before interpretation; a bounded reading is a designed outcome. On a public LendingClub dataset (temporal 2013 to 2016 and cross-segment transfer), ranking is stable and oracle gaps are small; the clearest temporal signal is a prevalence and probability-scale mismatch that intercept-only diagnostic recalibration largely reduces, though its cause is not identifiable from the available release. The positive control responds only to larger injected shifts; subtler drift cannot be excluded. Mapping diagnostic patterns to governance actions is conceptual guidance, not validated here.

1. Introduction

Credit-risk models use proxy labels and face changing deployment contexts, while no single transfer metric distinguishes base-rate, probability-scale, and relationship change. The paper proposes a locked multi-signal audit protocol and finds bounded evidence of stable ranking with a temporal prevalence and probability-scale mismatch.

1. Introduction

  • The study treats the default outcome as an operational proxy for realized credit risk, not a direct measure of creditworthiness, and notes that proxy reliability may change over time.Deployment conditions also evolve as macroeconomic conditions, lending policies, and borrower behavior change.Source: §1 · p. 1
  • A single transfer metric cannot distinguish base-rate shift, probability-scale mismatch, and feature-label relationship change in proxy-labeled credit-risk prediction.Ranking metrics can understate probability-scale change, while proper scoring rules can absorb base-rate and scale differences.Source: §1 · pp. 1–2
  • The locked protocol combines five diagnostic layers, thresholds, and decision rules fixed before interpretation, making bounded conclusions an intended outcome.Its layers cover transfer performance, an oracle-gap probe, calibration, feature-label stability, and a synthetic positive control.Source: §1 · p. 2
  • Stable ranking can coexist with probability-scale drift that misprices risk, so ranking alone is insufficient evidence of safe deployment.The paper notes that credit scores affect access to housing, education, and working capital.Source: §1 · pp. 1–2
  • On one public LendingClub dataset, ranking transfer was stable and oracle gaps were small; the clearest temporal signal was a prevalence and probability-scale mismatch largely reduced by diagnostic recalibration.The mismatch's cause is not identifiable from the available release, and the findings are bounded rather than proof that the conditional relationship is unchanged.Source: §1 · p. 2

2. Related Work

Prior work establishes concepts and methods for studying distribution shift, credit scoring, calibration, and proxy supervision. This paper positions its contribution as a pre-specified audit protocol for interpreting shift evidence when credit-risk labels are proxy-based and fixed.

2. Related Work

  • The study draws on research in dataset shift, credit scoring, calibration, and weak or proxy supervision, and contributes an audit protocol rather than a new model or adaptation method.Source: §2 · p. 2

2.1. Dataset shift and concept drift

  • Dataset-shift research supplies the vocabulary for covariate, prior-probability, and concept shift; this study uses complementary layers to characterize shift signatures rather than build a robust predictor.It distinguishes ranking transfer, probability-scale mismatch, marginal feature-label change, and target-learnable multivariate change.Source: §2.1 · p. 2

2.2. Credit scoring and shift in credit scoring

  • Prior credit-scoring work compares classifiers and studies distribution shift, while this study uses models as diagnostic instruments and holds proxy-label construction fixed.It audits which shift type the evidence supports rather than prescribing retraining or recalibration policies.Source: §2.2 · p. 2

2.3. Calibration and label shift

  • Calibration is distinct from ranking, and the protocol treats prevalence shift as one candidate explanation for temporal miscalibration rather than assuming it.Its calibration diagnostic uses target default-rate information in an oracle-style analysis, not a production correction.Source: §2.3 · pp. 2–3

2.4. Weak and proxy supervision

  • Work on weak and proxy supervision addresses label construction or formalizes supervision drift; this study instead audits whether fixed proxy supervision remains reliable under temporal and segment shift.Source: §2.4 · p. 3

2.5. Decision support, model-risk governance, and design science

  • The paper frames its audit as a decision-support artifact connecting model-risk governance with design-science research and its rigor-and-relevance approach.Source: §2.5 · p. 3

2.6. Positioning

  • Prior work leaves open how to interpret shift evidence with proxy-based labels and fixed target construction; the protocol separates stable ranking, probability-scale mismatch, and stronger relationship-drift evidence.Source: §2.6 · p. 3

3. Methods

The paper contributes a purposeful audit artifact and demonstrates its behavior, with conclusions bounded by a pre-committed design and an evaluation focused on diagnostic characterization.

3.1. The artifact as a design-science contribution

  • The contribution is a design-science artifact demonstrated on a real proxy-labeled task, not a generalization about credit markets.The LendingClub study instantiates the protocol and evaluates its behavior for rigor and relevance.Source: §3.1 · p. 3
  • Four principles organize the protocol: multi-signal disconfirmation, pre-commitment, mechanism separation, and an explicit sensitivity boundary.The design distinguishes recalibration-addressable base-rate or probability-scale shifts from relationship change that would call for retraining.Source: §3.1 · p. 3
  • Evaluation tests whether the protocol's layers characterize shift signatures across temporal and cross-segment transfer, with a positive control providing internal sensitivity analysis.Source: §3.1 · p. 3
  • The evaluation does not test whether governance actions are correct organizational choices, which depend on materiality and costs the protocol does not model.The action mapping is conceptual guidance, not a validated decision rule.Source: §3.1 · pp. 3–4

3.2. Empirical setting and prediction target

The study examines a fixed binary proxy outcome in one public LendingClub dataset, comparing an in-domain baseline with temporal and cross-segment transfers whose prevalence and sample sizes differ.

3.2. Empirical setting and prediction target

  • The target is a binary operational proxy for realized credit risk, and its construction remains fixed throughout the analysis.The focus is whether feature-proxy-label relationships and predictive behavior remain stable enough for transfer.Source: §3.2 · p. 4
  • Pure covariate shift is not a standalone target; it matters only insofar as it affects the protocol's five diagnostic layers.Source: §3.2 · p. 4

3.3. Transfer settings

  • The locked settings include an in-domain held-out-fold baseline, 2013-to-2016 temporal transfer, and transfer from 2013 debt-consolidation to 2013 credit-card loans.The settings differ in sample size and observed proxy-label prevalence, which conditions interpretation.Source: §3.3 · p. 4

3.4. Models and evaluation metrics

The evaluation uses three model families and combines ranking, class-imbalance, probability-error, and calibration measures to assess model behavior.

3.4. Models and evaluation metrics

  • Three model families—logistic regression, random forest, and histogram gradient boosting—span a linear model and two nonlinear tree-based models.Random forest runs across several seeds; the other learners use a fixed seed.Source: §3.4 · p. 4
  • Models are trained on a source context and evaluated on a target context.The passage specifies this setup alongside the seed plan.Source: §3.4 · p. 4
  • AUROC measures ranking, average precision addresses class imbalance, and the Brier score measures probability-scale error.Calibration uses equal-frequency-bin ECE and a logistic-regression intercept and slope on clipped model logits.Source: §3.4 · p. 4

3.5. Layered diagnostic protocol

The protocol layers complementary diagnostics because no single transfer measure identifies the mechanism of change. Each layer has a defined interpretive boundary, and the layers jointly inform rule-based synthesis.

3.5. Layered diagnostic protocol

  • The layered protocol treats its five diagnostics as complementary inputs to rule-based synthesis, not statistically independent confirmation.Several layers can respond to the same prior, base-rate, or label-shift structure.Source: §3.5 · p. 4

3.5.1. Primary transfer performance.

  • Transfer performance reports AUROC, average precision, and Brier score in target contexts, but cannot by itself identify the mechanism of change.It examines ranking and probability-scale behavior together.Source: §3.5.1 · p. 4

3.5.2. Oracle-gap diagnostic.

  • The oracle gap compares a source-trained model with a target-trained oracle on the same held-out target fold to probe target-learnable relationship change.The shared test fold removes test-set variation, but training-context, sample-size, and model-capacity differences remain.Source: §3.5.2 · p. 4

3.5.3. Calibration diagnostic.

  • Calibration diagnostics distinguish ranking from probability-scale mismatch using ECE and calibration intercept and slope, then test a target-rate-matching logit intercept shift.Because recalibration uses target labels, it is an oracle-style counterfactual for separating prevalence or scale mismatch from deeper relationship change, not a deployable procedure.Source: §3.5.3 · p. 4

3.5.4. Feature-label stability diagnostic.

  • Feature-label stability measures per-feature marginal association shifts against random-split noise, but cannot establish full stability of P(y|x).The Spearman measure is descriptive sensitivity checking, not the locked band metric; the locked metric is univariate AUC strength.Source: §3.5.4 · pp. 4–5

3.5.5. Positive-control sensitivity.

  • The positive control injects weak, medium, and strong relationship shifts to test diagnostic sensitivity, not validate the protocol.It relabels 5%, 15%, and 30% of target rows, changing 1.5%, 4.6%, and 9.2% of labels, respectively; dti_n and fico_n are the perturbation features.Source: §3.5.5 · p. 5

3.6. Rule-based multi-signal synthesis

Pre-specified rules synthesize the diagnostic layers into bounded readings rather than independent confirmation. Their thresholds were locked before official experiments, while governance-use phrases are conceptual guidance.

3.6. Rule-based multi-signal synthesis

  • Rules A–G combine performance, oracle gap, feature-label stability, and positive-control response into diagnostic readings; multiple applicable readings are reported.The rules were committed before official experiments and were not edited after outputs were seen; governance-use phrases were added conceptually, not as locked rules.Source: §3.6 · p. 5
  • AUROC drop and oracle gap are stable or small below 0.02, borderline from 0.02 to 0.05, and degraded or large at 0.05 or more.These are heuristic diagnostic anchors, not thresholds calibrated to portfolio cost.Source: §3.6 · p. 5
  • Feature-label instability is low at no more than twice noise, moderate from two to three times, and high at three times or more; calibration is poor if ECE10 exceeds 0.05, |intercept| exceeds 0.5, or slope falls outside [0.8, 1.25].Recalibration fixes most calibration error when ECE10 drops by at least 50% to at most 0.05.Source: §3.6 · p. 5
  • Positive-control detection requires an oracle gap at least 0.05 above the real temporal gap and a monotone weak-to-strong increase.Rules E and F gate evidential interpretation, while Rule G gives a bounded-evidence reading when the pattern is stable and the control works.Source: §3.6 · p. 5

3.7. Artifact availability

The protocol implementation, locked specifications, run manifests, and result tables are publicly available, with manifests documenting execution and input provenance.

3.7. Artifact availability

  • The protocol, locked settings, run manifests, and result tables are publicly available through the project repository and Zenodo archive.The specifications were committed before the first official run.Source: §3.7 · p. 5
  • Stage manifests record commands, execution commits, package versions, and input-file SHA-256 hashes; the raw analysis-file fingerprint recomputes from the file used.These records support reproduction and applying the protocol unchanged to new proxy-labeled settings.Source: §3.7 · p. 5
  • Table 2 presents the locked A–G decision-rule set, while text after its semicolons is conceptual governance guidance added for the paper.The governance phrases were not part of the locked rules.Source: Table 2 · p. 6, §3.7 · p. 6

4. Results

Across temporal and cross-segment transfer, ranking remained stable and oracle gaps showed no large advantage; the clearest temporal signal was a prevalence and probability-scale mismatch reduced by diagnostic recalibration. Feature-label instability was mostly low to moderate, and the positive control responded clearly only to strong injection.

4.2. Oracle-gap diagnostic

  • Temporal AUROC oracle gaps ranged from 0.008343 to 0.016399 and cross-segment gaps from −0.007268 to 0.006090; none reached the large threshold of 0.05.Supporting average-precision and Brier gaps were also small and did not suggest a qualitatively different picture from the AUROC analysis.Source: §4.2 · p. 6

4.3. Calibration diagnostic

  • Temporal raw ECE10 ranged from 0.071088 to 0.081410, falling to 0.005148 to 0.029311 after diagnostic recalibration; Figure 1 pairs values by model and seed.All temporal rows were poorly calibrated and met the locked criterion that recalibration fixes most error; this oracle-style recalibration is not deployable. Random-forest slope patterns also appeared in-domain, so were treated as model-specific rather than drift evidence.Source: §4.3 · pp. 6–7, Figure 1 · p. 7

4.1. Primary transfer performance

  • Mean AUROC ran from 0.612 to 0.659 across transfer settings, with no positive degradation of 0.02 or more; temporal Brier scores worsened by about 0.047 to 0.049 versus in-domain.Cross-segment Brier scores were slightly better than in-domain; Table 3 reports primary transfer performance across locked settings.Source: §4.1 · p. 6, Table 3 · p. 7

4.4. Feature-label stability

  • The locked numeric feature-label instability ratio was 2.4895 (moderate/borderline) temporally and 1.7518 (low) cross-segment; both categorical ratios were low.Categorical ratios were 1.4343 temporally and 0.8578 cross-segment; a higher descriptive Spearman-based temporal ratio of 4.8307 was a caveat, not the locked band metric.Source: §4.4 · p. 7

4.5. Positive-control sensitivity

  • Every model and seed’s oracle gap rose monotonically across injected shift levels; strong injection cleared the descriptive threshold for random forests, not logistic regression, while boosting stayed just below.The aggregate strong-level margin was 0.0547 when random-forest seeds counted as separate rows, but 0.0484 with equal weight per model family; these are descriptive summaries, not a locked decision rule.Source: §4.5 · p. 7, Figure 2 · p. 8

4.6. Multi-signal synthesis

  • Jointly, locked diagnostics found no strong evidence of large detectable relationship drift: temporal results were closest to Rule B, cross-segment to Rule-A-style stability, with an overall Rule-F-like sensitivity boundary.The positive control responded clearly only to strong injection and only for random-forest models, without a binary decision. Nearby cutoff checks left the qualitative reading unchanged, though some band labels moved.Source: §4.6 · pp. 7–8

5. Discussion

The locked diagnostics show no strong evidence of large detectable relationship drift in the specified transfers, while the observed temporal signature cannot be attributed to a specific mechanism. The discussion treats the bounded evidence as decision support, not validated governance rules.

5.1. Principal finding and what the protocol adds

  • The locked diagnostics show no strong evidence of large detectable relationship drift under the specified transfer settings, but do not establish that drift is absent.The protocol is an auditing framework, not a universally validated detector; its positive control is a descriptive sensitivity curve, not formal power analysis.Source: §5.1 · p. 8

5.2. Interpretation of the LendingClub evidence pattern

  • Convergence across layers sharing a failure mode is weaker evidence because calibration, Brier, and oracle-gap signals can all respond to prior shift.The oracle-gap and feature-label layers have different failure modes, making their agreement more informative.Source: §5.2 · p. 8
  • The temporal signature is observable, but the release lacks information needed to attribute it to a specific data-generating mechanism.Without contractual term, loan-resolution status, or outcome snapshot date, the prevalence difference cannot be decomposed into population change, maturity-related selection, cohort effects, or a mixture; origination-month checks were inconclusive.Source: §5.2 · p. 8, §5.4 · p. 9

5.3. Implications for credit-risk decision support and model-risk governance

  • For governance, the conceptual mapping would evaluate deployable recalibration before retraining or escalation, alongside monitoring and targeted label collection.The diagnostic recalibration uses target-context labels and is not deployable; the governance mapping is unvalidated and real actions depend on costs, regulatory, and operational factors.Source: §5.3 · pp. 8–9
  • Probability-scale errors may affect pricing and approval thresholds even when aggregate discrimination appears stable, but this study makes no empirical fairness claim.The study does not conduct a subgroup-fairness analysis; the implications motivate monitoring.Source: §5.3 · p. 9

5.4. Limitations

  • The evaluation uses one public LendingClub dataset, so the absence of large-drift evidence should not be generalized to other lending settings.The feature-label diagnostic is marginal and univariate; recalibration is oracle-style, and positive-control sensitivity is model- and level-dependent rather than a binary detector.Source: §5.4 · p. 9
  • The positive control responds only at strong injected shifts for some models, leaving smaller or subtler relationship drift possible within the sensitivity boundary.The oracle-gap layer is bounded by model capacity, feature information, label noise, sample size, and locked thresholds; low base discrimination also makes subtle drift harder to detect.Source: §5.4 · p. 9

5.5. Future work

  • Future work should apply the locked protocol to additional datasets and domains, with pre-registered dataset selection.Proposed extensions include richer models, a release with loan term and resolution status, and validation of the diagnostic-to-governance mapping.Source: §5.5 · p. 9

6. Conclusion

The paper presents a locked, multi-signal audit protocol for proxy-labeled credit-risk prediction and demonstrates it across temporal and cross-segment transfers on one public LendingClub dataset. The bounded results indicate limited positive-control sensitivity and an observed temporal probability-scale component, without identifying its mechanism or ruling out subtler drift.

6. Conclusion

  • The paper contributes a locked, multi-signal audit protocol demonstrated on one public LendingClub dataset across temporal and cross-segment transfer.The artifact is framed as design science.Source: §6 · p. 10
  • Positive-control sensitivity appeared only at strong injection and for some models; the temporal setting showed a Rule B-style prevalence and probability-scale component.Cross-segment behavior was consistent with Rule-A-style stability, while Rule D was not triggered under the locked thresholds.Source: §6 · p. 10
  • The protocol characterizes the observed signature but does not attribute its mechanism; smaller or subtler relationship drift remains possible.The release cannot separate population change from maturity-related selection or other cohort effects.Source: §6 · p. 10
Loading 2609.16102v1…