PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Self-Supervised Pretraining Help Tabular Models? A Study of Label Scarcity and Missing Data

Sahand Mazrouei

arXiv:2608.24381v1cs.LG

TL;DR

Tabular self-supervised learning remains insufficiently evaluated under scarce labels and missing features. This paper benchmarks mask-and-recover pretraining across diverse classification tasks, finding nuanced, variable benefits that are strongest on clean datasets and suggest robustness to test-time missingness.

  • Problem

    The interaction between tabular SSL reconstruction objectives, native missingness, label scarcity, and shifted test-time missingness remains an open empirical question.

  • Method

    The study evaluates a mask-and-recover pretrained tabular encoder against scratch training and classical baselines across 14 diverse classification tasks.

  • Results

    SSL is competitive with tree ensembles, but gains vary substantially across tasks; improvements are most reliable on clean datasets and test-time missingness robustness is suggestive rather than significant.

  • Takeaways & Limitations

    Tabular SSL benefits are dataset-dependent, motivating adaptive designs rather than assuming imputation objectives universally help natively incomplete data.

  • Takeaways & Limitations

    On datasets with high native missingness, additional artificial masking during pretraining can occasionally cause the model to underfit available signal.

Abstract

from arXiv · show

Self-supervised learning (SSL) has emerged as a promising approach for tabular data, yet its efficacy under extreme label scarcity and test-time missingness remains under-explored. In this paper, we evaluate a mask-and-recover SSL pretraining objective against training from scratch and classical baselines across 14 diverse classification tasks. First, while SSL outperforms training from scratch on average and remains competitive with state-of-the-art tree ensembles (achieving ~0.8954 AUC vs. Random Forest's 0.9015 at 10% labels), the SSL-vs-scratch gains exhibit high inter-task variance and lack significance (p = 0.626 at both 5% and 10% labels). Second, contrary to the hypothesis that missing-value imputation objectives universally benefit datasets with native missingness, SSL yields the most reliable improvements on clean datasets, while frequently degrading performance on datasets with high inherent missingness. Third, despite this training variance, SSL-pretrained models achieve a higher average AUC than scratch-trained models under both test-time missingness completely at random (MCAR) injection (+0.0245 AUC, positive on 11 of 14 tasks) and structured missingness shifts (MNAR, +0.0418 AUC, positive on 8 of 14 tasks), though neither difference remains statistically significant after Holm-Bonferroni correction for multiple comparisons (adjusted p = 0.118 and p = 0.518, respectively). Fourth, comparing our mask-and-recover objective against three established tabular SSL baselines (VIME, SCARF, SubTab) under an identical encoder architecture, we find no significant difference from any of them (adjusted p = 0.459, p = 1.000, p = 1.000), indicating our findings reflect general properties of tabular SSL rather than idiosyncrasies of one particular pretext task.

1 Introduction

The introduction frames tabular SSL as a response to label scarcity and missing data, then evaluates its benefits, robustness, component sensitivity, and objective-specificity under realistic constraints. Across these tests, SSL is competitive but its gains are dataset-dependent and generally not statistically distinguishable from alternatives.

  • Motivation: Tabular tasks often favor tree-based ensembles, while label scarcity and missing data make robust deep-learning approaches especially challenging.These challenges motivate evaluating SSL beyond fully observed, clean datasets.
  • Evaluation design: The study compares an SSL-pretrained encoder with the identical architecture trained from scratch and classical baselines across OpenML classification tasks.Experiments vary labeled fractions from 1%, 5%, 10%, and 20% and inject missingness at test time.
  • Label scarcity: Within roughly 0.006 AUC-ROC of Random Forest at 10% labels, SSL remains highly competitive, but its gains over scratch vary substantially across tasks and are not statistically significant.The result highlights strong aggregate performance without a reliable universal advantage over scratch training.
  • Missing-data interaction: SSL improves classification most consistently on clean datasets, whereas performance on datasets with high native missingness is highly dataset-dependent.Although reconstruction objectives inherently learn imputation, native missingness does not universally amplify SSL benefits.
  • Test-time robustness: Under +30% MCAR injection and MNAR shifts, SSL-pretrained models degrade more gracefully than scratch-trained models, but the advantage is not significant after multiple-comparison correction.The introduction presents robustness as a consistent average tendency rather than a statistically confirmed effect.

2 Related Work

Tabular learning remains dominated by tree-based models, motivating specialized deep architectures and self-supervised objectives to address their data hunger. Missing-data research spans MCAR, MAR, and MNAR mechanisms, with both classical and learned imputation approaches.

  • Deep learning for tabular data: Tree-based models such as XGBoost, LightGBM, and Random Forest remain dominant for tabular data, partly because deep models struggle with uninformative features and unsuitable inductive biases.
  • Deep learning for tabular data: Specialized architectures including TabNet and FT-Transformer adapt attention-based deep learning to tabular datasets.
  • Self-supervised learning: VIME, SCARF, and SubTab adapt self-supervised learning through feature recovery, contrastive corruption, and reconstruction from partial row views.
  • Missing-data handling: Missing data is categorized as MCAR, MAR, or MNAR, with approaches ranging from mean imputation, K-Nearest Neighbors, and MICE to MissForest and GAIN.

3 Background and Problem Setup

The study frames tabular classification with mixed numeric and categorical features, labels available only for training examples, and missingness represented by binary masks. It evaluates models using test AUC-ROC under controlled labeled-data fractions and train-only preprocessing.

  • Problem setup: The task is supervised classification on N examples with d features, including numeric features x_n, categorical features x_c, and labels y available only for a subset.
  • Problem setup: Missing entries are represented by binary masks m^(i), with 1 denoting a missing categorical feature entry.
  • Experimental protocol: Preprocessing and pretraining are fit only on the official OpenML training split and never on test data.
  • Experimental protocol: Labeled subsets comprise 1%, 5%, 10%, or 20% of the training set, with three seeds controlling subset sampling and fine-tuning while SSL pretraining is shared per task.
  • Evaluation: Test AUC-ROC is the primary metric, using standard two-class scoring for binary tasks and one-vs-rest macro-averaging for multiclass tasks.

4 Method

The method uses a dual-branch tabular encoder that explicitly represents missingness and fuses numeric and categorical features into a 128-dimensional representation. SSL pretraining masks, corrupts, and recovers inputs using type-specific reconstruction losses and representation-consistency regularization before fine-tuning.

  • Encoder architecture: Binary mask vectors mark missing numeric and categorical entries, which are processed alongside features by the dual-branch Tabular Encoder.The numeric branch concatenates features with masks and applies a two-layer MLP.
  • SSL pretraining: ρ = 0.3 corruption masks replace masked numeric values with 0.0 and categorical values with a dedicated [MASK] token during SSL pretraining.The pretraining paradigm generates two augmented views of each corrupted input.
  • SSL pretraining: Reconstruction combines masked numeric MSE and categorical CE, while cosine-distance consistency enforces invariant representations across views.When both feature types are masked, Lrecon = 0.5(Lnum + Lcat), with categorical features weighted equally rather than tokens.
  • Training procedure: 50 epochs of unlabeled SSL pretraining precede 30 epochs of supervised fine-tuning with AdamW, learning rate 10^-3, weight decay 10^-4, and batch size up to 128.The batch size is capped by the number of available labeled examples.

5 Experiments

Experiments evaluate the approach across 14 diverse classification tasks spanning different training sizes and native missingness rates. Comparisons include scratch training and classical baselines, with robustness tested under normal, MCAR, and structured MNAR missingness.

  • Experimental setup: 14 diverse OpenML classification tasks provide evaluation across datasets with substantially varying training sizes and native missingness rates.
  • Baselines: The approach is compared with the identical neural architecture trained from scratch using only labeled data.
  • Baselines: Classical baselines at 10% labels include Logistic Regression, Random Forest, XGBoost, and LightGBM.
  • Missingness robustness: Robustness is evaluated under normal test conditions, MCAR with 30% uniformly random feature missingness, and MNAR masking of numeric values above each column’s 70th percentile.
  • Statistical analysis: Classifier differences across datasets are summarized by mean ∆ and tested with Wilcoxon signed-rank tests, using Holm-Bonferroni correction across four primary hypotheses at α = 0.05.

6 Results

SSL’s benefits are modest and inconsistent in clean label-scarce settings, but pretrained models show larger average gains under test-time missingness. Neither missingness robustness nor comparisons with alternative SSL objectives establish statistically significant superiority, and multi-seed ablations expose substantial variance.

  • Label scarcity: +0.0074 average AUC over scratch at 5% labels and +0.0055 at 10% labels, but neither gain is significant.Wilcoxon p = 0.6257 at both fractions; Holm-adjusted p = 1.000.
  • Label scarcity: 0.9145 average SSL AUC falls below the 0.9173 scratch baseline at 20% labels, concentrating pretraining’s benefit in the label-scarce regime.The shrinking advantage crosses zero by 20% labels.
  • Native missingness: SSL improves most reliably on clean datasets, whereas high inherent missingness produces highly variable effects that frequently degrade performance.This contradicts the hypothesis that mask-and-recover objectives universally benefit datasets with native missingness.
  • Test-time missingness: +0.0245 AUC under MCAR +30% missingness and +0.0418 AUC under MNAR shift favor SSL over scratch, but Holm-adjusted p-values are 0.118 and 0.518.The MCAR raw p-value is 0.0295, while the MNAR raw p-value is 0.1726; neither comparison remains significant after correction.
  • Ablations: A mean ∆AUC of +0.0031 for removing consistency regularization does not replicate a single-seed improvement across all four native-missingness tasks.The effect helps Tasks 3021 and 146800 but hurts Tasks 2079 and 14954; group masking also shows no reliable direction (mean ∆AUC = +0.0004, p = 1.0).
  • SSL objective comparison: Adjusted p-values of 0.459, 1.000, and 1.000 show no significant difference between our objective and VIME, SCARF, or SubTab under an identical encoder.Mean differences are VIME −0.0024, SCARF +0.0011, and SubTab +0.0041.

7 Discussion

SSL helps most under severe label scarcity and on originally clean datasets, while its robustness to structured missingness is suggestive rather than statistically significant. Ablation effects are likewise inconsistent across tasks and indistinguishable from zero after averaging multiple pretraining seeds.

  • SSL improvements peak at 5% labels, shrink with more labels, and reverse sign by 20% labels.
  • The strongest and most reliable SSL gains occur on originally clean datasets, challenging universal benefits from native missingness.
  • +0.0418 AUC under structured MNAR shifts is positive on 8 of 14 tasks, but remains nonsignificant after Holm-Bonferroni correction (adjusted p = 0.518).The mask-and-recover objective may improve robustness by encouraging reliance on redundant predictive structures across features.
  • Across three pretraining seeds per variant, consistency regularization, group masking, and mask-aware pretraining showed task-varying effects with aggregate deltas statistically indistinguishable from zero.Initial ablations had suggested consistency regularization was harmful on datasets with native missingness, but that effect did not survive seed averaging.

8 Limitations and Broader Impact

The study demonstrates promising robustness under label scarcity and test-time degradation but has important empirical limitations, unresolved mechanisms, and interpretability concerns. Its potential impact is substantial in expensive-label domains, provided practitioners remain cautious about deep tabular models’ limitations.

  • Limitations: Self-supervised pretraining does not universally benefit all tabular data distributions, particularly datasets with high inherent missingness.
  • Limitations: Near-ceiling tasks contribute low-variance deltas to paired statistical tests, diluting the effective sample despite nominally using n = 14.
  • Limitations: The root cause of the Missing vs. Clean Paradox remains unresolved, as the proposed mask-visibility mechanism could not be confirmed.
  • Broader Impact: Robust modeling under extreme label scarcity could benefit domains requiring expensive expertise, including medical diagnostics and fraud detection.
  • Broader Impact: Zero-shot robustness to test-time MNAR sensor degradation may reduce catastrophic production failures, but deep tabular models remain less interpretable.

9 Conclusion

The study provides a nuanced evaluation of tabular self-supervised pretraining, emphasizing that its success depends on dataset context and robust multi-seed validation. It calls for dataset-adaptive designs and broader native-missingness benchmarks to clarify uncertain component effects.

  • Conclusion: The evaluation establishes a nuanced understanding of when and why self-supervised learning succeeds on real-world tabular data.The study examined tabular deep learning under extreme label scarcity and test-time missingness across diverse OpenML datasets.
  • Conclusion: Seed-averaged ablations found no reliable effects from view-consistency regularization, group masking, or mask-aware pretraining on native-missingness tasks.An initially observed harmful consistency-loss effect did not replicate across independent pretraining seeds.
  • Conclusion: Multi-seed evaluation is essential for small-sample tabular SSL ablation studies because single-seed findings may not replicate.The consistency-loss result changed after averaging across independent pretraining seeds.
  • Future Work: A larger and more diverse pool of native-missingness datasets could provide statistical power to determine whether consistency loss and group masking have genuine effects.The proposed future work targets four native-missingness tasks studied here and distinguishes small effects from inert components.

A Per-task Results and Robustness Table · B Data Dictionary and Context

The paper reports per-task AUC-ROC and robustness deltas for SSL versus scratch training across label-scarcity and test-time missingness settings. It evaluates 14 diverse OpenML datasets whose domains, feature types, and missingness properties are documented individually.

  • A Per-task Results and Robustness Table: Table 3 reports per-task AUC-ROC for SSL-pretrained and scratch models at 1%, 5%, 10%, and 20% labeled data.Metrics are means over 3 random seeds and rounded to 3 decimals, while statistical tests use full-precision values.
  • A Per-task Results and Robustness Table: Table 4 reports ∆AUC (SSL−Scratch) at 10% labels under normal evaluation, MCAR (+30% missingness), and MNAR structured shift.The table uses seed 42.
  • B Data Dictionary and Context: The benchmark comprises 14 OpenML datasets spanning diverse domains, feature dimensionalities, and missingness properties.A brief context description is provided for each dataset.
  • B Data Dictionary and Context: The dataset contexts include ecological, medical, industrial, and biological tasks with reported missingness ranging from 1.7% to 22.2%.Examples include Eucalyptus at 4.3%, Sick at 22.2%, Cylinder Bands at 5.3%, and MiceProtein at 1.7% missingness.
  • B Data Dictionary and Context: Anneal is evaluated on a preprocessed numerical subset with 0.0% missingness despite heavy structural missingness in its raw categorical variables.The task predicts the annealing process of steel.
  • B Data Dictionary and Context: Other documented benchmarks cover cytological cancer scoring, credit approval, software defects, income prediction, and image-derived recognition tasks.These include Breast-W, Credit-Approval, JM1, Adult, Letter, Pendigits, Diabetes, Spambase, and Vehicle.
  • B Data Dictionary and Context: The remaining contexts include diagnostic prediction, email spam classification, and 3D vehicle-silhouette classification.These correspond to Diabetes, Spambase, and Vehicle, respectively.

C Baseline Implementation Details … D.2 Tree-Based Ensembles

The paper prioritizes reproducibility through fixed preprocessing, deterministic classical baselines, and uniform deep-model configurations across 14 datasets. Tree ensembles use 100 estimators, while the deep encoder and SSL setup remain dataset-agnostic.

  • C Baseline Implementation Details: Categorical features were isolated, cast to strings, and ordinal-encoded; Logistic Regression used max_iter=1000 for convergence.These preprocessing and optimization choices addressed float coercion errors, dimensionality, and dataset variance.
  • C Baseline Implementation Details: Random Forest used n_estimators=100, n_jobs=-1, and default Gini splitting, while XGBoost used eval_metric=’logloss’ and default boosting hyperparameters.LightGBM used verbose=-1 during evaluation across the 14 datasets.
  • C Baseline Implementation Details: All classical baselines used random seed 42, unlike the deep models’ 3-seed protocol, and relied on simple statistical rather than learned imputation.The passage explicitly distinguishes this setup from methods such as GAIN and MissForest.
  • D.1 Deep Tabular Encoder and SSL Objective: The deep tabular architecture was identical across all 14 datasets without dataset-specific tuning, isolating the effect of the self-supervised objective.The dual-branch MLP was deliberately simpler than attention-based tabular architectures and learned numerical-feature embeddings.
  • D.1 Deep Tabular Encoder and SSL Objective: The encoder used two-layer, 128-dimensional numeric and fusion MLPs with Layer Normalization and dropout p = 0.1, plus E = 32 categorical embeddings.Unknown test-time categories were mapped to index 0.
  • D.1 Deep Tabular Encoder and SSL Objective: Pretraining used AdamW with Learning Rate = 10−3, Weight Decay = 10−4, Batch Size = 256, Grad. Clip 1.0, and OneCycleLR.The consistency weight was λ = 1.0, with ablation to 0.0 and results averaged over 3 independent pretraining seeds.
  • D.2 Tree-Based Ensembles: All tree ensembles used 100 estimators; Random Forest specified max_features = ’sqrt’, bootstrap = True, and unrestricted depth, while XGBoost and LightGBM retained listed default-style configurations.XGBoost used learning_rate = 0.3 and max_depth = 6; LightGBM used learning_rate = 0.1, num_leaves = 31, and boosting_type = ’gbdt’.
Loading 2608.24381v1…