PaperlayerView exact revision on arXiv

Source-linked AI summary

CRISP: Scalable Importance-Stratified Coresets for Imbalanced Tabular Learning

Hardhik Mohanty, Indrayana Rustandi, Mohamadreza Sheibani

arXiv:2609.26962v1cs.LG

TL;DR

Repeated GBDT training on large imbalanced tabular datasets is costly, and existing coreset methods can lose accuracy at high majority-class reduction. CRISP allocates a negative-class budget across proxy-score quantile strata and weights selected examples. It retains 99.7% of production full-data monthly AP at 95% negative reduction, with leading mean AP across CriteoPrivateAds reduction rates and mixed Sparkov results.

  • Problem

    Repeated training on large imbalanced tabular datasets is costly, while existing coreset methods can lose accuracy when most majority examples are removed.

    Source: §1 · p. 1

  • Method

    CRISP uses a proxy GBDT to allocate a negative-class budget across score quantile strata and applies clipped inverse-inclusion weights to selected negatives.

    Source: §1 · p. 2, §8 · p. 8

  • Results

    At 95% negative reduction, CRISP retains 99.7% of full-data monthly AP on production data; it has the highest mean AP at every tested CriteoPrivateAds rate, while Sparkov results are less uniform.

    Source: §1 · p. 2, §8 · p. 8

  • Takeaways & Limitations

    The experiments support importance-aware allocation when the negative budget is scarce, with evidence across production fraud and public advertising data.

    Source: §1 · p. 2, §8 · p. 8

    The evaluation does not establish end-to-end wall-clock speedup or universal dominance, and production hyperparameter variation weakens isolation of the selection method.

    Source: §7 · p. 8

Abstract

from arXiv · show

Large imbalanced tabular datasets make repeated gradient-boosted tree training expensive. Existing coreset methods often lose accuracy when most majority examples are removed. We present CRISP (Coreset Reduction via Importance-Stratified Pruning), a linear-time method that allocates a negative-class budget across quantile strata of a proxy-model score. Sample weights account for unequal inclusion probabilities. At 95% negative-class reduction on a production fraud dataset, CRISP trains on approximately 1.70M of 25M rows and retains 99.7% of full-data Average Precision. This is a 93.2% reduction in total training rows. On public CriteoPrivateAds, CRISP has the highest mean Average Precision at each tested rate from 90% to 99.4% majority reduction. Sparkov results are mixed at lower rates, but CRISP has the highest mean at 99.2% and 99.4%. Ablations identify budget allocation and inverse-propensity weighting as the main sources of the production-dataset gain.

1 Introduction

Repeated GBDT training makes large imbalanced tabular datasets costly, while uniform removal treats informative boundary negatives and redundant negatives alike. CRISP allocates a negative-class budget across proxy-score strata and weights selected negatives; reported results show strong production retention and leading CriteoPrivateAds mean AP across tested rates.

1 Introduction

  • Uniform sampling removes informative and redundant negatives at the same rate, despite the rare positives and boundary negatives carrying more learning signal.In fraud and conversion prediction, most negative rows are described as easy and many occupy dense interior regions.Source: §1 · p. 1
  • The pipeline retains all positives and applies its reduction budget only to negatives, so negative-class reduction is not the same as total-row reduction.One-shot selection is designed for reuse across downstream training runs, favoring a lightweight proxy fit and a small number of data passes.Source: §1 · p. 1
  • CRISP uses quantile strata to preserve broad score coverage, allocates more budget to strata with larger proxy scores, and applies clipped inverse inclusion weights.The proxy is separate from the downstream model; quantiles, aggregation, scoring, and sampling remain distributed.Source: §1 · pp. 1–2
  • On the proprietary production dataset of about 25 million rows, CRISP retains 99.7% of full-data monthly AP after removing 95% of negatives, using approximately 6.8% of original rows.Source: §1 · p. 2
  • CRISP has the highest mean AP at every tested rate on CriteoPrivateAds, while Sparkov results are less uniform and sensitive to outliers.The datasets span production fraud, public advertising, and synthetic fraud settings.Source: §1 · p. 2
  • The contributions include estimator analysis distinguishing an unbiased unclipped inverse-propensity estimator from the biased clipped estimator used for stable GBDT training.The work also reports production-scale and cross-domain evidence, component ablations, and row-count tradeoffs distinct from end-to-end selection cost.Source: §1 · p. 2

2 Related Work

Prior coreset approaches trade off geometric coverage, model awareness, and selection cost. CRISP is positioned as a reusable, linear-pass approach for highly reduced imbalanced tabular GBDT training.

Imbalanced instance selection

  • Classical instance-selection methods use local geometric rules, but distance computations become expensive and less informative as dimension and sample count grow.LSH approximates neighborhoods to reduce cost, while its selection criterion remains independent of the downstream model.Source: §2 · p. 2

Proxy and geometry-based coresets

  • Proxy and geometry-based coresets use coverage, proxy scoring, gradient approximations, boundary reconstruction, or performance constraints, but most cited evaluations are outside large-scale tabular settings or at moderate pruning rates.FCTR reports a three- to ten-fold selection speedup over gradient-based competitors.Source: §2 · p. 2

Training dynamics and sample importance

  • Training-dynamics and sample-importance scores can identify useful or atypical examples, but checkpoint trajectories or multiple models can cost as much as downstream training.Examples include forgetting [25], dataset cartography [23], GraNd and EL2N [18], TracIn [19], and SAMIS.Source: §2 · p. 2

Gradient and validation matching

  • Gradient- and validation-matching methods offer direct optimization targets, but pairwise similarities, repeated gradient evaluations, or iterative updates limit use on tens of millions of rows.Small-downsample comparisons do not establish whether a selector can run on production-scale data.Source: §2 · p. 2

High reduction and tabular models

  • CCS preserves easy and moderate examples through uniform difficulty-stratum allocation, while CoreTab adapts training-dynamics ideas to GBDTs using costly repeated prediction passes.Uniform allocation does not distinguish strata when the final budget is very small.Source: §2 · p. 2

Connection to survey sampling

  • CRISP applies inverse inclusion weights after importance-based allocation across proxy-score strata, differing from CCS allocation and direct importance sampling while avoiding pairwise gradient similarities.The stated practical target is reusable selection for imbalanced tabular GBDTs at negative-class reductions above 90%. Clipping means the deployed estimator is not exactly unbiased and carries no claimed general variance guarantee.Source: §2 · pp. 2–3

3 Methodology: CRISP

CRISP retains positives and reduces only negatives, using proxy-model scores to allocate a negative-row budget across quantile strata. Its distributed selector uses a fixed-size proxy fit and a small number of passes over the training table.

3.1 Problem Formulation

  • CRISP retains every positive and sets the target negative budget from the negative-class reduction rate, making expected training size the retained positives plus the negative budget.The retained positive fraction creates a floor on target-training size.Source: §3.1 · p. 3

3.2 Budget Interpretation

  • At 95% negative reduction, production data removes about 93.2% of all rows because positives make up roughly 1.9% of the experimental training table.At rates above 99% on Sparkov and Criteo, retained positives can outnumber retained negatives.Source: §3.2 · p. 3

3.3 Proxy Model and Importance Scores

  • A lightweight proxy GBDT scores negative rows for selection; its score ranks fraud risk rather than estimating a calibrated population fraud rate.The proxy is trained on at most N_sub negatives and all positives, and is not reused as the downstream classifier.Source: §3.3 · p. 3

3.4 Budget Allocation and Sampling

  • CRISP partitions negatives into equal-count score quantiles and allocates each stratum a budget proportional to its mean score, with integer rounding and redistribution when needed.The production implementation samples proportionally to scores within strata, whereas Sparkov and CriteoPrivateAds sample uniformly within strata; Bernoulli sampling makes realized sizes vary by seed.Source: §3.4 · p. 3
  • Selected negatives receive clipped inverse-inclusion weights, while positives receive weight one for downstream GBDT training.The distributed workflow scores rows in batches, aggregates stratum counts and scores, and broadcasts the compact allocation table rather than collecting the full score vector.Source: §3.4 · p. 3, §3.5 · pp. 3–4

3.5 Distributed Implementation

  • Proxy fitting is capped independently of dataset size, while scoring, approximate quantiles, aggregation, and sampling use a small constant number of distributed passes, yielding O(n) selection cost.The implementation records target and realized negative counts and sample-weight ranges; memory is distributed across Spark workers.Source: §3.5 · p. 4

4 Design Rationale and Estimator Properties

CRISP separates score-region coverage from budget allocation, then uses inverse-inclusion weighting to estimate full-data empirical loss. The weighting result is conditional on the realized design and fixed parameter; clipping and score-based allocation are practical choices with stated limits.

4.1 Coverage Under a Small Budget

  • Equal-count quantiles preserve representation across score regions concentrated near zero, while allocation gives higher-score strata more budget and keeps lower-score strata eligible.Each stratum begins with a comparable candidate count, and low and moderate strata retain nonzero inclusion probabilities.Source: §4.1 · p. 4

4.2 Weighting and Estimator Scope

  • With positive inclusion probabilities and conditional expected selection rates matching π_i, inverse-probability weights make estimated loss unbiased for full-data empirical risk at fixed θ.This result uses the realized sampling design and does not require independent selections.Source: §4.2 · p. 4
  • Clipping negative weights at w_max is a stability choice, not an unbiased correction, and the fixed-parameter loss result does not guarantee the same trained model or AP retention.The proposition concerns empirical loss at a fixed θ, not the downstream model after optimization.Source: §4.2 · p. 4

4.3 Allocation as a Practical Heuristic

  • CRISP substitutes mean proxy-model score for unavailable within-stratum variation; it resembles Neyman allocation when score and loss variation correlate, but is not the exact solution.Source: §4.3 · p. 4
  • Production and Criteo follow the predicted pattern as the negative budget shrinks, while Sparkov is less consistent, so the allocation rule is not universally optimal.The prediction is that importance-aware allocation separates from uniform allocation as k− shrinks.Source: §4.3 · p. 4

5 Experiments

The experiments evaluate CRISP on production fraud, synthetic fraud, and advertising conversion data using Average Precision. Results are strongest on the production dataset and CriteoPrivateAds, while Sparkov is mixed at lower reduction rates.

5.1 Datasets and Protocol

  • The evaluation covers financial transactions, synthetic fraud, and advertising conversion, using temporal splits for production and Criteo and multiple seeds for the benchmarks.Production uses three seeds, while Sparkov and Criteo use five; target hyperparameters can differ because each method’s selected data informs the zero-shot suggestion.Source: §5 · p. 4, §5.1 · pp. 4–5

5.2 Production Dataset Results

  • At 95% negative reduction, CRISP retains 99.7% of full-data APmonth versus 94.2% for Random and 94.1% for CCS, using 1.70M rows (93.2% total-row reduction).The expected CRISP training set is about one-third the size of the Random and CCS sets at their 80% reduction comparison point.Source: §5.2 · p. 5, Table 4 · p. 6

5.4 Sparkov

  • On Sparkov, CRISP has the highest mean at 99.2% and 99.4% reduction but loses to Random and CCS at 99%; Random leads at 95%.At 99%, CRISP’s standard deviation is largest because one run falls to 0.6647; Random’s 95% mean is lifted by a run reaching 0.9629.Source: Table 6 · p. 6, §5.4 · p. 6

5.3 Ablation

  • At 95% reduction, Q=10 retains 99.0% versus 95.2% for Q=1; clipped IPW retains 99.2% versus 97.4% without IPW.The ablation jobs were separate from Table 4, so their reference values should not be compared as if they were the same runs.Source: §5.3 · p. 5, Figure 3 · p. 7

5.5 CriteoPrivateAds

  • Across 90%–99.4% negative reduction, CRISP has the highest mean AP; its lead over Random grows from 0.0034 AP to 0.0074 AP.At 99.4% reduction, about 1.16M rows remain, or 1.37% of the full training split.Source: §5.5 · pp. 6–7, Table 7 · p. 7

5.6 Target-Model Transfer

  • At 95% negative reduction, CRISP retains 98.9% of monthly and 98.2% of weekly full-data AP with XGBoost, ahead of Random and CCS.These existing runs provide a second target-model comparison, but remain limited to one dataset.Source: §5.6 · p. 7

5.7 Compute Interpretation

  • Figure 4 plots AP against target-training rows, not wall-clock time, and therefore does not establish end-to-end speedup.CRISP adds proxy-model fitting and distributed scoring before target training.Source: §5.7 · p. 7

5.8 Cross-Dataset Analysis

  • Across datasets, CRISP’s advantage is clearest as negative budgets shrink, but Sparkov remains outlier-sensitive and the tested allocation is not established as universally optimal.Fixed-hyperparameter studies and end-to-end timing measurements remain necessary for cleaner causal and systems comparisons.Source: §5.8 · p. 7

6 Reproducibility

The paper documents public-benchmark evaluation materials while withholding the proprietary production data and implementation details. Its row-count tradeoff estimates target-model training cost, not end-to-end runtime.

6 Reproducibility

  • The paper documents public-benchmark splits, methods, configurations, reduction rates, seeds, metrics, aggregate results, and per-run values in the appendix.No external artifact release is pledged.Source: §6 · p. 7
  • The proprietary production dataset, implementation, configurations, feature definitions, logs, run metadata, and records are not released, and absolute production AP values are withheld.Production results are normalized by the corresponding full-data model.Source: §6 · p. 7
  • Figure 4 estimates target-model training cost from reported class counts and the keep-all-positives rule, excluding selection overhead.Historical hardware details were not consistently preserved, so the paper reports target-training rows rather than wall-clock time.Source: Figure 4 · p. 8, §6 · p. 8

7 Limitations and Ethics

The evaluation has methodological and scope constraints, and the paper identifies further subgroup checks as necessary before deployment. Ethical reporting also distinguishes an operational fraud label from customer intent.

7 Limitations and Ethics

  • Retaining every positive requires a sufficiently small positive class to leave a useful negative budget; production and public benchmarks also use different within-stratum sampling rules.Bernoulli sampling varies realized coreset size, while clipping introduces bias.Source: §7 · p. 8
  • The row-count tradeoff omits selection and calibration costs, and varying FLAML suggestions weaken isolation of the selection method.Evaluation uses three or five seeds without paired tests, subgroup metrics, or fixed-hyperparameter replication; claims are strongest for observed AP retention, not universal dominance or deployment cost.Source: §7 · p. 8
  • The production dataset is handled under institutional data-governance controls, and transactions and user identifiers are not released.Direct identifiers are removed from public datasets during preprocessing.Source: §7 · p. 8
  • The payment-reversal label is an operational outcome, not evidence of customer intent, and stable aggregate AP may still accompany altered subgroup representation or false-positive rates.Before deployment, compare subgroup recall, calibration, and manual-review burden with the full-data model.Source: §7 · p. 8

8 Conclusion

CRISP targets highly imbalanced tabular learning by stratifying negative examples with a proxy GBDT and weighting unequal-probability samples. Results are strongest on the production dataset, while the paper limits its claims about generality and runtime.

8 Conclusion

  • A proxy GBDT assigns negatives to score quantiles, whose mean scores set stratum budgets; inverse-inclusion weighting addresses distribution shift from unequal sampling.The procedure uses a fixed-size proxy fit and distributed passes, without pairwise gradients or predictions across many training checkpoints.Source: §8 · p. 8
  • At 95% negative-class reduction, CRISP retains 99.7% of full-data monthly AP while using about 1.70M of 25M rows.CriteoPrivateAds has the same mean ordering across five reduction rates; Sparkov is less stable, and outlier runs caution against claiming general superiority from one rate.Source: §8 · pp. 8–9
  • The XGBoost experiment adds evidence that selected production rows transfer across two tree implementations.The supplied conclusion does not report a numerical result for this experiment.Source: §8 · p. 9
  • The experiments support importance-aware allocation when the negative budget is scarce, but establish neither a universal variance guarantee nor an end-to-end wall-clock speedup.The paper calls for common within-stratum rules, fixed target hyperparameters, actual timing, repeated paired estimates, and subgroup checks.Source: §8 · p. 9

A Estimator Derivation

  • Conditional on the realized design, the estimator is unbiased for the full loss; unconditional unbiasedness follows by the law of total expectation.The argument uses linearity of expectation and does not require independent indicators.Source: A Estimator Derivation · p. 9
  • Clipping preserves an expected row contribution when inclusion probability is at least 1/w_max; below that threshold, it contributes π_i w_max times the loss.The derivation uses this difference from the full loss to obtain Equation 10.Source: A Estimator Derivation · p. 9
  • The production variant samples within strata in proportion to score, whereas Sparkov and CriteoPrivateAds use a constant Bernoulli fraction within each stratum.All three share the importance-based allocation.Source: A Estimator Derivation · p. 9

B Full Per-Run Experimental Results

The appendix reports relative per-run production results at the two highest reduction rates and provides per-run tables for Sparkov and CriteoPrivateAds.

B Full Per-Run Experimental Results

  • Table 8 reports relative per-run production results at the two highest reduction rates, followed by per-run Sparkov and CriteoPrivateAds tables.The supplied passage does not provide the table values or comparisons.Source: B Full Per-Run Experimental Results · p. 10

C Full Per-Run Sparkov Results

The Sparkov passages report per-run AP baselines and note that mean and median rankings differ because several methods have large outliers.

C Full Per-Run Sparkov Results

D Sparkov Robust Summaries

  • Table 10’s mean and median rankings differ because Random, CoreTab, and CRISP each have at least one large outlier.The table reports medians and interquartile ranges across the same five runs as Table 6.Source: D Sparkov Robust Summaries · p. 10

F Full Per-Run CriteoPrivateAds Results

The CriteoPrivateAds per-run results cover negative-class reduction rates and compare five seed runs with a single full-data reference run.

F Full Per-Run CriteoPrivateAds Results

Related papers

Loading 2609.26962v1…