Source-linked AI summary
SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
Dara Bahri, Heinrich Jiang, Yi Tay, Donald Metzler
TL;DR
Self-supervised contrastive learning has been successful in vision and language but remains underdeveloped for real-world tabular data. SCARF addresses this gap by forming views through random feature corruption and reports improved classification performance across supervised, noisy-label, and semi-supervised settings. Its results also show complementarity with existing strategies and robustness across several design choices.
Problem
Contrastive self-supervised methods have largely used domain-specific view generation, with little work applicable to real-world tabular data.
Method
SCARF pre-trains an encoder by contrasting original tabular examples with views formed by replacing randomly selected features using their empirical marginal distributions.
Results
SCARF improves classification accuracy across fully supervised, label-noise, and semi-supervised settings, outperforming alternative pre-training approaches and adding a 1-2% relative gain across the board.
Takeaways & Limitations
SCARF provides a simple, versatile tabular pre-training procedure that complements mixup, label smoothing, distillation, and dropout.
Takeaways & Limitations
Potential negative side effects include representations that reinforce biases present in the input data.
Abstract
from arXiv · showhide
Self-supervised contrastive representation learning has proved incredibly successful in the vision and natural language domains, enabling state-of-the-art performance with orders of magnitude less labeled data. However, such methods are domain-specific and little has been done to leverage this technique on real-world tabular datasets. We propose SCARF, a simple, widely-applicable technique for contrastive learning, where views are formed by corrupting a random subset of features. When applied to pre-train deep neural networks on the 69 real-world, tabular classification datasets from the OpenML-CC18 benchmark, SCARF not only improves classification accuracy in the fully-supervised setting but does so also in the presence of label noise and in the semi-supervised setting where only a fraction of the available training data is labeled. We show that SCARF complements existing strategies and outperforms alternatives like autoencoders. We conduct comprehensive ablations, detailing the importance of a range of factors.
1 INTRODUCTION
SCARF addresses the limited availability of domain-general self-supervised methods for tabular data by introducing contrastive views through random feature corruption. It is evaluated as pre-training for downstream classification and is reported to improve performance across supervised, noisy-label, and semi-supervised settings.
- Self-supervised learning leverages abundant unlabeled data to learn representations for downstream tasks when labels are costly.
- Existing contrastive methods generate views using domain-specific distortions, leaving few techniques applicable across domains and especially to tabular data.
- SCARF forms views by randomly selecting features and replacing them with draws from their respective empirical marginal distributions.
- SCARF is tested on real-world classification datasets and improves accuracy in fully supervised, label-noise, and semi-supervised settings.
- The method learns task-agnostic representations, while ablations examine design choices and hyperparameter stability.
2 RELATED WORKS
Prior work spans domain-specific self-supervised methods, contrastive losses, and tabular corruption-based approaches. SCARF uses marginal feature randomization for views and contrasts with related masking, dropout, and reconstruction strategies.
- Vision and language methods use generated images, token replacement, masking, and related domain-specific transformations for self-supervised learning.
- SCARF pre-training updates encoder f and head g by minimizing the contrastive loss with stochastic gradient descent.
- InfoNCE is a popular contrastive loss, and SCARF finds it effective for its framework.
- Yao et al. mask random features in a correlated manner and apply dropout to categorical features, whereas SCARF independently samples feature values from empirical marginals.
- VIME uses the same tabular corruption technique but pre-trains an encoder to recover the corruption mask and original input.
3 SCARF
SCARF creates corrupted tabular views by replacing randomly selected features with values sampled from empirical marginals, then trains representations with an InfoNCE contrastive objective. The learned encoder is fine-tuned with a classification head.
- For each unlabeled mini-batch example, SCARF replaces a uniformly sampled fraction of features with draws from each feature’s empirical marginal distribution.The marginal distribution is uniform over the values observed for that feature in the training dataset.
- The original and corrupted examples pass through encoder f and pre-training head g to produce normalized representations z(i) and ˜z(i).The pre-training head ℓ2-normalizes outputs onto the unit hypersphere.
- InfoNCE pulls representations from the same example together and pushes representations from different examples apart.Networks f and g are optimized with SGD.
- Fine-tuning attaches classification head h to encoder f and optimizes cross-entropy while updating both networks.
- Early stopping uses validation InfoNCE loss tracked on a static set of corrupted validation pairs.
4 EXPERIMENTS
Experiments evaluate SCARF across real-world tabular classification datasets using multiple training settings, baselines, and ablations. SCARF generally outperforms alternative pre-training methods while remaining robust across corruption strategies and key hyperparameters.
- Experimental setup: Experiments use 69 OpenML-CC18 classification datasets with shared 70%/10%/20% train/validation/test splits and repeated trials.The evaluation covers fully labeled data, reduced labeling, and corrupted training labels.
- Baselines and evaluation: SCARF is compared with label smoothing, dropout, Mixup, autoencoders, self-training, and tri-training alongside no-pretraining controls.The study also evaluates win matrices and relative accuracy improvements across datasets.
- Label noise: SCARF boosts baseline performance when 30% of training labels are corrupted and improves label-noise solutions such as Deep k-NN.The experiments keep validation and test labels uncorrupted while corrupting a random subset of training labels.
- Predictive performance: SCARF outperforms other pre-training techniques, improves existing methods, and adds a 1-2% relative gain across baselines.The reported gains include improvements to Mixup, label smoothing, distillation, and dropout.
- Corruption ablations: Marginal sampling generally outperforms alternative corruption strategies and remains invariant to feature scaling without additional hyperparameters.The comparison covers z-score and min-max scaling.
- Hyperparameter ablations: Performance remains stable for corruption rates of 50%-80%, temperature changes, and batch sizes above 128 provide no significant improvement.The paper recommends defaults of 60% corruption and temperature 1.
5 CONCLUSION
SCARF addresses limited progress in self-supervised learning for tabular data by learning representations effective for downstream classification, including with limited labels or label noise. The paper identifies potential bias reinforcement as a side effect and a direction for future research.
- SCARF learns representations effective for downstream classification in the general tabular setting.
- SCARF remains effective when labeled data is limited or labels contain noise.
- Representations learned by SCARF may reinforce biases present in input data.
- Mitigating input-data bias reinforcement during training is identified as future research.
A APPENDIX
The appendix uses the same hyperparameters as the main experiments unless otherwise noted.
- The appendix generally reuses the main experimental hyperparameters unless otherwise noted.
A.1 RESULTS FOR DATA-LIMITED EXPERIMENTS
In the experiment with only 25% of training labels available, SCARF outperforms the other methods.
- SCARF outperforms the other methods when only 25% of training labels are available.
A.2 PRE-TRAINING LOSS CURVES
SCARF’s training and validation losses decrease rapidly initially, then decline more slowly until early stopping, while training loss remains noisy because corruption is stochastic.
- Both SCARF training and validation losses decrease rapidly at first, then diminish more slowly until early stopping.
- SCARF training loss has high variance because the corruption method is stochastic.
A.3 ABLATIONS CONTINUED
The ablation results are evaluated using accuracy with all labeled data available, enabling comparison of design choices under the fully supervised setting.
- Accuracy is the evaluation metric for the ablation results.
- The ablations use 100% of the labeled data.
- These experiments assess SCARF design choices in a fully labeled setting.
IMPACT OF BATCH SIZE AND CORRUPTION RATES
Figure 5 examines how batch size and corruption rate affect SCARF in the fully labeled, noiseless setting.
- Figure 5 evaluates the impact of batch size on SCARF performance.
- Figure 5 evaluates the impact of corruption rate on SCARF performance.
- Both factors are studied with all labels available and without label noise.
IMPACT OF CORRUPTION STRATEGIES
The ablations identify effective corruption and training choices for SCARF, including one-view corruption, per-example sampling, marginal replacement, and vanilla InfoNCE. They also compare pre-training, validation choices, scaling, and standard baselines across settings.
- Impact of corruption strategies: Corrupting one view outperforms corrupting both, while per-example feature-index sampling outperforms using one shared index set across a mini-batch.At a 60% corruption rate, corrupting both views leaves less shared information between them and makes the contrastive task harder.
- Impact of corruption strategies: Bernoulli selection of a variable number of corrupted features performs roughly the same as the constant-count selection in Algorithm 1.
- Impact of corruption strategies: Replacing corrupted features with entries from one randomly drawn training example for the whole mini-batch performs worse than empirical-marginal replacement.
- Impact of corruption strategies: SCARF’s marginal-sampling corruption generally outperforms alternative strategies under z-score and min-max scaling.The strategy is also competitive with mean scaling and is invariant to feature scaling.
- Training strategy and evaluation: SCARF pre-training outperforms co-training, additive-noise autoencoders, and supervised-only augmentation, while InfoNCE error validation degrades downstream performance.The comparisons include fully supervised, label-noise, semi-supervised, and absolute-accuracy evaluations across datasets and baselines.
- Loss and hyperparameter choices: Vanilla InfoNCE outperforms Uniform-Align and shows no benefit from Barlow Twins, while equal weighting is best within Uniform-Align.A softmax temperature of 1 performs well and is recommended.