PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Fair Tabular Foundation Models

Patrik Kenfack, Jesse C. Cresswell, Anthony L. Caterini, Samira Ebrahimi Kahou, Ulrich Aïvodji

arXiv:2608.14211v1cs.LGcs.AI

TL;DR

Fairness in tabular foundation models remains underexplored, while existing fairness methods do not readily fit in-context learning. FairTFM addresses this with fairness-aware pretraining and improves fairness by 32–75% relative to the strongest TFM baseline at accuracy costs of 2–11%.

  • Problem

    Fairness properties of tabular foundation models remain underexplored, and existing fairness methods do not readily extend to single-pass in-context learning.

  • Method

    FairTFM synthesizes fairness tasks during pretraining and uses adversarial sensitive-attribute prediction with a gradient reversal layer for fair single-pass inference.

  • Results

    Fairness improves by 32–75% relative to the strongest TFM baseline at accuracy costs of 2–11% across 132 fairness tasks.

  • Takeaways & Limitations

    Fairness can be incorporated directly into tabular foundation model pretraining while maintaining competitive predictive performance in in-context learning.

  • Takeaways & Limitations

    The pretraining prior may not capture all semantic and societal structure of real sensitive attributes, and FairTFM does not dominate specialized baselines on every task or metric.

Abstract

from arXiv · show

Tabular Foundation Models (TFMs) have emerged as leading methods for tabular predictive tasks, leveraging in-context learning to predict on new data without task-specific training. Despite the increased use of TFMs in high-stakes decision-making, their fairness properties remain largely unexplored. In this work, we incorporate fairness constraints directly into TFM training, enabling fair predictions in a single forward pass. Our approach addresses two key challenges: limited access to sensitive attributes in training data, and the incompatibility of existing fairness techniques with the in-context learning paradigm. We propose FairTFM, a scalable training strategy based on synthetic fairness tasks and a fairness-aware architecture using a gradient reversal layer, which encourages the model to learn representations invariant to sensitive attributes. Experiments on 132 fairness tasks show consistent improvements in fairness while maintaining competitive accuracy.

1 Introduction

FairTFM addresses the largely unexplored fairness risks of tabular foundation models by integrating fairness-aware training into in-context learning. It combines scalable synthetic fairness tasks with an adversarial transformer architecture and improves fairness while preserving competitive predictive performance.

  • Motivation: TFMs can produce biased outcomes despite strong predictive performance, while classical fairness methods do not readily fit single-forward-pass in-context learning.TFMs adapt to new tasks using only a few labeled examples, without task-specific optimization.
  • Method: FairTFM generates synthetic fairness tasks by randomly designating an input feature as a categorical sensitive attribute during pretraining.Repeating this process for every sampled dataset supports scalable fairness-aware training across diverse tasks.
  • Evaluation: 120 fairness tasks from ACS PUMS and 12 additional tasks from six fairness benchmarks evaluate demographic parity, equal opportunity, and equalized odds.The benchmark tasks test generalization, with comparisons against strong baselines.
  • Results: FairTFM consistently improves fairness metrics while maintaining competitive predictive performance across the evaluated tasks.The reported fairness evaluation covers demographic parity, equal opportunity, and equalized odds.
  • Method: FairTFM extends transformer-based TFMs with a fairness-aware architecture and adversarial learning to enable fair predictions in a single forward pass.The architecture handles sensitive attributes whose identities change across tasks and that are masked at inference.

2 Related work

Prior fairness methods commonly intervene before, during, or after model training, but their assumptions are poorly aligned with frozen-predictor in-context learning. Recent tabular foundation models prioritize predictive accuracy, while FairPFN addresses causal fairness and provides only partial group-fairness benefits at a predictive cost.

  • Fairness: Fairness methods typically act through data pre-processing, training-time constraints, or output post-processing, often assuming training-pipeline access or calibrated outputs.These assumptions are less compatible with in-context learning, where a pretrained model serves as a frozen predictor at inference.
  • Tabular foundation models: Recent tabular foundation models, including TabPFN, TabDPT, and TabICL, show strong competitiveness on tabular prediction but largely emphasize accuracy rather than biased outcomes.
  • Tabular foundation models: FairPFN incorporates fairness into TFM pretraining using a causal fairness objective, contrasting with this work’s focus on statistical group fairness.
  • Tabular foundation models: FairPFN provides partial group-fairness benefits but incurs a predictive cost, is Pareto-dominated by FairTFM, and lacks a mechanism to control the fairness–utility trade-off.

3 Training fair tabular foundation models

FairTFM pretrains on synthetic fairness tasks that randomly designate and hide sensitive attributes, then uses a shared transformer with adversarial representation learning for fair in-context predictions. Its joint objective balances target accuracy against sensitive-attribute invariance without requiring sensitive attributes at inference.

  • Fairness task synthesis: Fairness tasks are synthesized by randomly designating one dataset feature as sensitive, removing it from inputs, and discretizing continuous attributes into categorical groups.This self-supervised task generation exposes pretraining to diverse forms of group-dependent bias.
  • Model architecture: FairTFM processes feature, label, and sensitive-attribute tokens through a shared transformer and predicts both the target label and sensitive attribute with separate MLP heads.The backbone alternates row- and column-wise attention to support in-context learning along both axes.
  • Fairness mechanism: A gradient reversal layer adversarially trains the shared representation to remove information predictive of the sensitive attribute.Unlike separately trained adversaries, this formulation avoids fitting a new adversary for each sampled fairness task during pretraining.
  • Inference setting: At inference, FairTFM replaces unavailable query sensitive attributes with a fixed learnable parameter, so sensitive attributes are never directly provided for query predictions.The encoder is trained so that providing the query sensitive attribute would not be particularly fruitful.
  • Pretraining objective: The joint cross-entropy objective induces a fairness-accuracy trade-off controlled by λ when target labels and sensitive attributes are correlated.Each pretraining step creates a new fairness task, splits it into context and query sets, masks query labels and sensitive attributes, and updates the shared model.

4 Results

Across 120 ACS fairness tasks and 12 additional non-ACS tasks, FairTFM consistently improves fairness while maintaining competitive accuracy. Its fairness–accuracy trade-off is controlled by λ, and its learned representations suppress sensitive information while preserving task-relevant predictive content.

  • Fairness-accuracy trade-off across tasks: FairTFM variants trace a broad fairness–accuracy Pareto frontier, with larger λ values producing fairer but less accurate predictions.The evaluated λ values are 0.1, 0.7, 10, and 25.
  • Fairness-accuracy trade-off across tasks: TabICLv2 and TabPFNv2.5 achieve the highest average accuracy but occupy the fairest-inferior, most-unfair region of the Pareto frontier.FairTFM variants occupy the low-unfairness end while retaining competitive or stronger accuracy than classical baselines such as LR and KNN.
  • Generalization beyond ACS PUMS tasks: FairTFM generalizes beyond ACS tasks, improving fairness without a commensurate loss in AUCROC while preserving the same accuracy–fairness pattern.The non-ACS evaluation covers 12 tasks.
  • Performance dynamics during training: Both FairTFM variants achieve a fairness advantage over nanoTabPFN early in pretraining and retain it throughout optimization.The comparison tracks accuracy, demographic parity difference, and equalized odds difference for λ ∈ {0.7, 25}.
  • Comparison with fairness-constrained classical models: Against fairness-constrained classical baselines, FairTFM-0.7 improves relevant metrics by 20–33% and FairTFM-25 by 79–82% on the non-ACS tasks.EG reduces XGBoost’s AUCROC from 0.803 to about 0.70, whereas FairTFM-0.7 remains at 0.803.
  • Fairness-enhancing representation learning: FairTFM embeddings reduce sensitive-attribute AUC toward random guessing while largely preserving task-relevant information.Sensitive-attribute AUC reaches 0.542 at λ = 25, while task-relevant AUC changes from 0.834 to 0.805 at λ = 0.7.

5 Conclusion · A Additional Related Work

FairTFM integrates synthetic fairness tasks and adversarial sensitive-attribute prediction into tabular foundation model pretraining, enabling single-pass predictions with improved group-fairness metrics and competitive accuracy. The paper also identifies limitations in synthetic pretraining coverage and metric-specific competitiveness, and situates the method among fairness interventions, in-context learning, tabular foundation models, and fairness-aware TFMs.

  • 5 Conclusion: FairTFM is presented as a pretraining framework that provides group fairness while preserving single-pass inference for in-context learning.Its central contribution is integrating fairness into tabular foundation model pretraining rather than applying task-specific updates at inference.
  • 5 Conclusion: Synthetic fairness tasks and a gradient reversal layer couple label prediction with adversarial sensitive-attribute prediction during pretraining.The architecture encourages fairness-aware representations through adversarial sensitive-attribute prediction.
  • 5 Conclusion: 132 fairness tasks consistently improve demographic parity, equal opportunity, and equalized odds while maintaining competitive accuracy.The evaluation covers 120 ACS-based fairness tasks and 12 additional tasks from six non-ACS benchmarks.
  • Limitations and future work: FairTFM’s pretraining relies only on TabICL’s prior generation, which may not capture all semantic and societal structure of real sensitive attributes.This limitation concerns the scale and diversity of synthetic pretraining data relative to sensitive attributes encountered in downstream deployments.
  • Limitations and future work: FairTFM remains broadly competitive but does not dominate specialized fairness-aware baselines on every metric or every task.The conclusion explicitly qualifies the empirical results rather than claiming universal superiority.
  • A Additional Related Work: Fairness-aware learning methods are commonly categorized as pre-processing, in-processing, or post-processing according to where intervention occurs.Examples include label massaging and fair representation learning for pre-processing, objective modification for in-processing, and output constraints for post-processing.
  • A Additional Related Work: For in-context learning, FairTFM encourages fairness during pretraining so internal representations support fairer predictions without task-specific parameter updates.The method combines aspects of in-processing and representation learning for inference-time use.
  • A Additional Related Work: Tabular foundation models evolved from specialized supervised architectures toward pretrained models that generalize across tasks through in-context learning.TabPFN established synthetic-task pretraining for strong zero-shot prediction, while TabPFNv2.5, TabICLv2, and TabDPT advanced efficiency, scale, or related capabilities.

B Datasets · B.1 Prediction Task Details · B.2 Task Construction

The benchmark uses folktables tasks derived from ACS PUMS and evaluates fairness across varied prediction problems, sensitive attributes, and states. Fairness tasks combine each base prediction task with Gender, Age, or Race across eight states to broaden evaluation beyond individual configurations.

  • B Datasets: The evaluation datasets and task-construction procedures are described as the basis for the benchmark.
  • B.1 Prediction Task Details: The benchmark is constructed from folktables, which derives its data from the ACS PUMS.
  • B.1 Prediction Task Details: Five binary prediction tasks commonly used in prior fairness work are considered.
  • B.1 Prediction Task Details: ACSIncome predicts whether annual income exceeds $50,000 among individuals older than 16 who worked at least one hour weekly and earned at least $100.
  • B.2 Task Construction: Fairness tasks are created by combining each base prediction task with Gender, Age, and Race and data from eight states.
  • B.2 Task Construction: Varying prediction problems, sensitive attributes, and states tests fairness across different label distributions, demographic compositions, and regional contexts.
  • B.2 Task Construction: Evaluating the full Cartesian product reduces the risk that fairness improvements are driven by only a few favorable settings.

C Fairness metrics

The paper evaluates group fairness through demographic parity, equalized odds, and equal opportunity, which quantify disparities in binary predictions across demographic groups. Experiments report empirical group gaps, where smaller values indicate fairer behavior and zero denotes perfect parity.

  • Fairness criteria: The fairness analysis focuses on disparities in binary classifier behavior across groups defined by the sensitive attribute S.Predictions are represented as ˆY = f(X), with binary ground-truth labels Y and sensitive attributes S.
  • Fairness criteria: Demographic parity (DP) requires equal positive-prediction rates across demographic groups.It compares P(ˆY = 1 | S = 0) and P(ˆY = 1 | S = 1).
  • Fairness criteria: Equalized odds (EOD) requires equal true-positive and false-positive rates across groups for each ground-truth label.The criterion enforces parity in P(ˆY = 1 | S, Y = y) for y ∈ {0, 1}.
  • Fairness criteria: Equal opportunity (EOP) requires parity only in true-positive rates and is the y = 1 special case of equalized odds.Its condition compares positive-prediction rates between groups among examples with Y = 1.
  • Empirical metrics: Experiments use empirical disparity gaps for DP, EOD, and EOP; smaller values indicate fairer behavior, with zero representing perfect parity.These gaps provide interpretable summaries of group-level disparities and are standard for binary-classification fairness evaluations.

D Model Architecture and Hyperparameters … E.2 Percent improvement over the strongest baselines

FairTFM combines a lightweight fairness-aware transformer with synthetic-task pretraining and reports a fairness–accuracy trade-off across benchmark tasks. The reported results indicate that unconstrained TFMs are most accurate but least fair, while FairTFM variants occupy the fairer end of the performance frontier.

  • D Model Architecture and Hyperparameters: The backbone uses six transformer layers, six attention heads, and 192-dimensional embeddings and feed-forward representations.It is based on the nanoTabPFN architecture and includes separate encoders for features, targets, and sensitive attributes.
  • D Model Architecture and Hyperparameters: Pretraining uses 300,000 synthetic tabular datasets, each containing 150 datapoints, six features, and two classes.Training uses batch size 32, with query labels masked and query sensitive attributes replaced by a learned mask token.
  • D Model Architecture and Hyperparameters: The fairness-aware variants vary λ, which controls the adversarial sensitive-attribute objective and the fairness–accuracy trade-off.Optimization uses Schedule-Free AdamW with learning rate 1 × 10^-4 and no weight decay.
  • E.1 Aggregated results on the 120 ACS tasks: Across the aggregated results, unconstrained TFMs are most accurate but least fair, whereas FairTFM variants trace the fair end of the Pareto front.This agreement holds when predictive performance is measured by either accuracy or AUCROC.
  • E.1 Aggregated results on the 120 ACS tasks: Table 2 aggregates accuracy, AUCROC, and fairness metrics across 120 benchmark tasks as means ± standard deviations over three random seeds.It compares FairTFM models with unconstrained TFMs and classical baselines.
  • E.2 Percent improvement over the strongest baselines: Table 3 measures each FairTFM variant’s relative change against TabPFNv2.5, XGB, and LR using averages from Table 2.Positive fairness values indicate lower DP, EOD, or EOP, while negative accuracy and AUCROC values indicate lower predictive performance.

E.3 Performance dynamics during training

During pretraining, FairTFM develops progressively fairer behavior, with λ controlling the final fairness–accuracy operating point. Higher λ achieves lower DP and EOD gaps but reduces accuracy relative to lower-λ FairTFM and nanoTabPFN.

  • E.3 Performance dynamics during training: The fairness–accuracy trade-off induced by λ is visible throughout training and defines distinct operating regimes for the model variants.The curves average checkpoints over 10 real-world fairness tasks, with shaded regions showing one standard deviation across tasks.
  • E.3 Performance dynamics during training: FairTFM with λ = 0.7 maintains consistently lower DP and EOD gaps than nanoTabPFN while reaching very similar final accuracy.Lower DP and EOD indicate fairer behavior.
  • E.3 Performance dynamics during training: FairTFM with λ = 25 attains by far the lowest unfairness on both DP and EOD metrics.Its accuracy remains below that of nanoTabPFN and FairTFM with λ = 0.7.
  • E.3 Performance dynamics during training: Representations become progressively less informative about the sensitive attribute during training, while λ provides direct control over the final fairness–accuracy operating point.Fairness is evaluated using DP and EOD, alongside accuracy.

E.4 Fair representation learning: protocol details and additional results · E.5 Additional Exponentiated Gradient results

E.4 establishes a representation-learning protocol comparing FairTFM with task-specific classical preprocessing, while E.5 evaluates FairTFM against Exponentiated Gradient baselines across additional fairness–performance views. FairTFM generally offers competitive trade-offs, with its strongest relative advantage appearing under AUCROC evaluation.

  • E.4 Fair representation learning: protocol details and additional results: The fairness condition is equivalent to P(Z | Y, S) = P(Z | Y) and I(Z; S | Y) = 0, motivating fairer downstream models trained on Z.This defines representation invariance conditionally on the target Y.
  • E.4 Fair representation learning: protocol details and additional results: FairTFM is compared with Correlation Remover and Learning Fair Representations, two sklearn-compatible fairlearn preprocessing methods that require task-specific representation fitting.CR removes linear dependence on sensitive attributes, whereas LFR maps inputs to latent representations.
  • E.4 Fair representation learning: protocol details and additional results: The ACSIncome Alabama evaluation uses gender, race, and age as sensitive attributes, with each task split into 80% training and 20% test data.CR and LFR fit maps on training data, while FairTFM uses training examples as context and test examples as queries.
  • E.4 Fair representation learning: protocol details and additional results: Downstream LR, RF, and XGB classifiers are trained on raw features and representations from CR, LFR, and FairTFM.CR and LFR fairness–accuracy parameters are swept over [0.01, 0.02, . . . , 0.1, 0.2, . . . , 1.0], while FairTFM checkpoints use Figure 2’s λ values and three random seeds.
  • E.4 Fair representation learning: protocol details and additional results: Across all three downstream models, FairTFM embeddings provide competitive fairness–accuracy trade-offs without task-specific representation-map fitting.CR is most effective for the linear LR model because it removes linear correlation.
  • E.5 Additional Exponentiated Gradient results: For race, FairTFM spans a wide and competitive frontier against EG-constrained baselines, particularly at the lower-unfairness end.For age, EG-based classical models achieve stronger accuracy–fairness trade-offs for EOD and EOP.
  • E.5 Additional Exponentiated Gradient results: Under AUCROC evaluation, EG’s randomized ensembles suffer a sharp drop, while FairTFM provides the best overall Pareto front across all three sensitive attributes.The comparison uses AUCROC instead of accuracy and attributes the EG decrease to trading calibrated scores for constraint satisfaction.

E.6 Evaluation beyond ACS PUMS tasks

Evaluation on 12 fairness tasks beyond ACS PUMS confirms the fairness–utility trends across diverse benchmarks. FairTFM variants substantially reduce disparities, with stronger fairness gains than on ACS at moderate predictive cost.

  • Benchmark scope: The evaluation adds widely used fairness benchmarks spanning different domains, prediction targets, and sensitive attributes beyond the 120 ACS PUMS tasks.These benchmarks define 12 additional fairness tasks.
  • Evaluation protocol: Results use the Section 4 evaluation protocol, average three random seeds, and report accuracy Pareto fronts, AUCROC views, and averaged metrics.The reported checkpoint is the last pretraining checkpoint for each FairTFM variant, making the summary conservative.
  • Relative performance: 39–50% fairness improvement: FairTFM-0.7 improves DP, EOD, and EOP against TabPFNv2.5 for a 5.2% accuracy and 3.0% AUCROC cost.Against XGB and LR, it improves fairness by 34–51% with no AUCROC cost, with AUCROC equal to 0.803 for all three models.
  • Overall findings: Unconstrained TFMs are most accurate but least fair, whereas FairTFM variants sharply reduce disparities at moderate predictive cost.This pattern matches the results observed on the 120 ACS tasks.
  • Relative performance: 82–89% fairness improvement: FairTFM-25 improves fairness across TabPFNv2.5, XGB, and LR at the fair end of the trade-off.The improvement is averaged over the 12 fairness tasks beyond ACS PUMS.

E.7 Fairness-aware baselines on the 12 non-ACS tasks

Across 12 non-ACS tasks, FairTFM is compared with Exponentiated Gradient-wrapped XGBoost under DP, EOD, and EOP constraints. All FairTFM variants improve targeted fairness metrics while achieving substantially higher predictive performance than EG ensembles.

  • Experimental setup: FairTFM is compared with XGBoost wrapped in Exponentiated Gradient across 12 non-ACS tasks and three fairness constraints: DP, EOD, and EOP.The fairness-violation tolerance ε is swept for each constraint.
  • Fairness results: Relative to best EG values—DP 0.062, EOD 0.083, EOP 0.070—FairTFM-0.7 improves by 21%/33%/20%, FairTFM-10 by 66%/70%/64%, and FairTFM-25 by 82%/82%/79%.The percentages correspond respectively to DP, EOD, and EOP.
  • Accuracy results: EG randomized ensembles reduce XGBoost’s AUCROC from 0.803 to roughly 0.69–0.70, while FairTFM achieves substantially higher predictive performance.The comparison reports this accuracy tradeoff alongside improvements on the targeted fairness metrics.
Loading 2608.14211v1…