PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Provably Private Classification with a Tabular Foundation Model

Talal Alrawajfeh, Cristiana Diaconu, Ossi Räisä, Sebastian Rodriguez Beltran, Yuan He, John Bronskill, Richard E. Turner, Antti Honkela

arXiv:2610.10068v1cs.LGcs.CR

TL;DR

Designing tabular foundation models around privacy, rather than adding privacy retrospectively to downstream training, is a central challenge. PrivTab embeds a private-summary mechanism in the foundation-model architecture and uses a single forward pass to fit each new sensitive dataset. It retains competitive predictive utility under privacy while reducing computational cost.

  • Problem

    How to design foundation models around privacy rather than adding privacy retrospectively to downstream training remains a research challenge.

    Source: Discussion · p. 8

  • Method

    PrivTab embeds a privacy mechanism in its foundation-model architecture and fits a new sensitive dataset by constructing a private summary in a single forward pass.

    Source: Discussion · p. 7

  • Results

    PrivTab has a positive AUC gap for most of 33 datasets at each of three plotted privacy levels, particularly against DP-MLP, and reduces fitting cost relative to dataset-specific private training.

    Source: Private classification across datasets · p. 5, Discussion · p. 7

  • Takeaways & Limitations

    PrivTab illustrates how foundation models can be designed around privacy from the outset while keeping access to sensitive data explicit and auditable.

    Source: Discussion · p. 8

    The study covers classification only, assumes one user per row, and requires preprocessing to be included in the privacy analysis.

    Source: Discussion · pp. 7–8

Abstract

from arXiv · show

Tabular data underpin prediction and decision-making in medicine, finance, government and science, but often contain sensitive individual-level information, creating a need for accurate prediction while preserving privacy. Traditional private learning provides formal privacy guarantees, but requires slow dataset-specific optimisation, suffers substantial utility loss under strong privacy, and is often difficult to apply correctly. Tabular foundation models adapt rapidly to new datasets, but existing models lack formal privacy guarantees, and are highly vulnerable to membership-inference attacks, limiting their use on sensitive data. Here we introduce PrivTab, an easy to use tabular foundation model for differentially private classification that embeds a privacy mechanism within its architecture. Pretrained on simulated datasets, PrivTab uses in-context learning to transform sensitive rows into compact, provably private summaries---effectively learning how to learn under privacy. PrivTab outperforms private linear and neural-network baselines under moderate-to-strong privacy, shows negligible membership leakage, maintains well-calibrated predictions under strong privacy, and reduces dataset fitting time by 10,000 times, requiring only a single forward pass. By combining formal privacy, speed, and easy of use, PrivTab brings recent advances in AI to applications where sensitive individual-level data have limited their adoption.

Principled and efficient privacy-aware in-context learning

PrivTab places a differentially private summary mechanism inside a tabular foundation model, so sensitive context data can be processed once and reused for downstream queries. Its design combines privacy-aware pretraining with a bounded, auditable attention mechanism.

Principled and efficient privacy-aware in-context learning

  • A compact DP summary of the sensitive context can be reused for downstream queries, making prediction independent of the original context size and adding no privacy loss after construction.The summary can be stored and queried indefinitely because downstream predictions rely solely on it.Source: Principled and efficient privacy-aware… · p. 3
  • Prior pretrained-model approaches such as DP-TabICL [Carey et al., 2024] rely on general-purpose LLM inference, requiring repeated inference per query row; DP-TabICL reports 0.14–1.11 seconds per row and evaluates only 10% of target rows on larger datasets owing to inference costs.Source: Principled and efficient privacy-aware… · p. 3
  • The evaluation targets dedicated private tabular methods that fit on CPU or GPU and support efficient CPU prediction, reflecting the stated computational regime.The paper notes that downstream users may have limited resources and rely on standard CPU hardware.Source: Principled and efficient privacy-aware… · p. 3
  • Pretraining across simulated tasks and privacy levels teaches PrivTab to account for privacy noise in predictions and uncertainty estimates.The paper contrasts this with standard private-learning pipelines, which do not provide this guarantee by default unless predictive uncertainty is explicitly modelled.Source: Principled and efficient privacy-aware… · p. 3
  • DP-MHCA produces a private summary whose sensitivity does not grow with context size, while keeping privacy-critical operations in a self-contained transformer module.The bounded cross-attention design aims to preserve attention expressiveness while bounding each record’s influence.Source: Principled and efficient privacy-aware… · p. 3
  • Lean 4 verification covers the sensitivity bound and adaptive Gaussian DP composition, while empirical audits found behaviour consistent with intended noise calibration.The formalisation does not verify the exact software execution.Source: Principled and efficient privacy-aware… · p. 3

Private classification across datasets

Across 33 TabArena classification datasets, PrivTab is more accurate in moderate-to-strong privacy regimes and has the highest aggregate Elo rating. Its fitting is far faster than dataset-specific private training, while its AUC advantage varies with dataset size and baseline.

Private classification across datasets

  • The benchmark evaluates 33 datasets across six privacy regimes and ten splits, using 80% of each split’s labelled rows as context and 20% as held-out targets.PrivTab uses variants specialised for small and large datasets; unlike the private baselines, it requires no dataset-specific optimisation.Source: Private classification across datasets · pp. 3–4
  • PrivTab is more accurate than DP-MLP and DP-LR in moderate-to-strong privacy regimes and achieves the highest aggregate Elo rating across datasets, privacy levels, and splits.The advantage is most pronounced in log loss under heavy noise; DP-MLP closes the gap only under weak privacy constraints.Source: Figure 2 · p. 4, Private classification across datasets · p. 4
  • PrivTab’s AUC advantage over DP-MLP diminishes as datasets grow, whereas its advantage over DP-LR generally grows with dataset size.PrivTab has a positive AUC gap for most of the 33 datasets at each of the three plotted privacy levels, particularly against DP-MLP.Source: Private classification across datasets · pp. 4–5, Figure 3 · p. 5
  • Approximately 10 milliseconds per context–target split is PrivTab’s median fitting time, more than four orders of magnitude faster than the minutes required by DP-MLP and DP-LR.Across the benchmark, DP-LR and DP-MLP required approximately 400 and 130 GPU hours, respectively, versus 30 GPU seconds for PrivTab.Source: Private classification across datasets · p. 5
  • On CPU, median prediction time for the full target set is 66.6 ms for PrivTab, 0.35 ms for DP-MLP, and 0.18 ms for DP-LR.These medians are computed across the 33 datasets after averaging timings within each dataset.Source: Private classification across datasets · p. 5

Private preprocessing in practice

The preprocessing case study tests whether PrivTab can retain useful performance when preprocessing is also private. Public column descriptions provide a way to infer clipping bounds without using sensitive values.

Private preprocessing in practice

  • The benchmark isolates prediction with common preprocessing, but in practice the statistics used to shift and scale sensitive columns must be privatised before release with the summary.The same statistics are used to transform query rows.Source: Private preprocessing in practice · p. 5
  • Clipping intervals bound numeric features by truncating values outside lower and upper bounds, limiting sensitivity of estimated means and variances for private preprocessing.The study inferred these intervals from public descriptions using an LLM, without supplying sensitive values to it.Source: Private preprocessing in practice · p. 6
  • At µ = 0.4, PrivTab achieved the highest Elo rating across five datasets, while LLM-derived and oracle bounds yielded similar AUC and log loss on four of five.The oracle comparison is only partially private; the LLM-derived bounds use public descriptions and avoid sensitive values when selecting clipping intervals.Source: Private preprocessing in practice · p. 6, Figure 4 · p. 6

Non-private models reveal membership

The record-level membership audit finds that PrivTab remains much closer to perfect privacy than non-private tabular foundation models on the evaluated clinical dataset. The study emphasizes vulnerable records rather than relying only on aggregate averages.

Non-private models reveal membership

  • The likelihood-ratio attack distinguishes context members from non-members using prediction confidence, which can be higher for queries on records used as context in unprotected TFMs.PrivTab and private-learning methods use DP mechanisms that limit how reliably membership can be inferred from prediction confidence.Source: Non-private models reveal membership · p. 6
  • On the Maternal Health Risk dataset, conservative lower confidence estimates of fitted attack accuracy approach 100% for the most vulnerable records under TabPFN v3 and TabICL v2, while PrivTab stays closer to perfect privacy.The authors note that empirical attacks illustrate practical relevance but cannot replace formal proofs.Source: Non-private models reveal membership · p. 7, Figure 5 · p. 7
  • The audit reports record-wise vulnerability because aggregate averages can hide highly vulnerable records.Figure 5a plots record-wise AUC distributions and Figure 5b shows ROC curves for the most vulnerable record at each displayed method or privacy level.Source: Non-private models reveal membership · p. 7, Figure 5 · p. 7

Discussion

PrivTab amortizes private fitting through a single-pass summary, while keeping sensitive-data access explicit and auditable. Its scope remains limited by classification-only evaluation and assumptions about rows, privacy strength, context size, and preprocessing.

Discussion

  • A single forward pass constructs the private summary, amortizing fitting across privacy levels without retraining while retaining competitive predictive utility under privacy.The approach places the privacy mechanism inside the pretrained foundation-model architecture rather than repeating dataset-specific private optimisation.Source: Discussion · p. 7
  • The summary creates an auditable privacy boundary: later predictions are post-processing, and the summary can be stored and shared for reuse.PrivTab is pretrained with the same privacy mechanism used at prediction, so it learns to predict in the presence of privacy noise.Source: Discussion · p. 7
  • The study covers classification, not regression, and assumes one user per row; extending guarantees to users contributing multiple rows is less straightforward than with DP stochastic optimisation.Source: Discussion · pp. 7–8
  • PrivTab has a smaller relative advantage when privacy is weak or context sizes are large, while preprocessing remains application-dependent and must enter the privacy analysis.Source: Discussion · p. 8
  • PrivTab illustrates designing foundation models around privacy from the outset, with inference through a private representation and explicit, auditable access to sensitive data.Source: Discussion · p. 8

Methods

The Methods section describes PrivTab’s differentially private context-to-summary mechanism and prediction from the released summary, alongside simulated-task pretraining, private-baseline selection, benchmark evaluation, and private standardisation. It also covers membership-inference auditing, verification of the private summary mechanism, and release of code and pretrained weights.

Differential privacy

  • Privacy is reported using µ-GDP, where smaller µ means stronger privacy, and computation using a released private summary incurs no additional privacy cost.Under replace-one adjacency, the Gaussian mechanism calibrates noise to function sensitivity; post-processing preserves the privacy guarantee.Source: Differential privacy · p. 9

Model architecture

  • The LBANP-style model embeds rows into 256-wide tokens, compresses context through 128 learned pseudo-tokens, and uses private cross-attention before query-to-summary prediction.Its summary module has three private cross-attention layers and two self-attention refinement layers; context labels are used for context embeddings, while query labels are withheld.Source: Model architecture · pp. 9–11
  • PrivTab’s summary mechanism releases the summary and predictions for any number of public queries with a µ-GDP guarantee under replace-one adjacency.Each of Lmain = 3 private cross-attention layers adds calibrated Gaussian noise; subsequent summary updates and predictions are post-processing.Source: Model architecture · pp. 10–11

Pretraining

  • PrivTab was pretrained on simulated TabICL tasks with noisy private summaries as inputs, optimizing target-label cross-entropy averaged over tasks and Gaussian summary noise.Two fixed-weight variants cover different context sizes and privacy levels; privacy noise is supplied explicitly when predictions are trained and made.Source: Pretraining · pp. 11–12

Private baselines

  • DP-LR and DP-MLP baselines used DP Adam with privacy loss composed across updates, while dataset-specific learning-rate selection accounted for candidate training and validation evaluations.Learning-rate selection used µsel = 0.5µ, with the final training budget chosen under GDP composition.Source: Private baselines · pp. 12–13

TabArena evaluation

  • The TabArena benchmark compared all methods on 33 datasets, ten 80/20 context–target splits, and six µ levels, evaluating AUC, log loss, and Elo rankings.The broad benchmark used common idealised standardisation; private preprocessing was evaluated separately.Source: TabArena evaluation · p. 13

Private preprocessing in practice

  • Private standardisation clips features using public bounds and releases noisy context sums and squared sums; its privacy budget composes with prediction or baseline fitting.The case study used µtotal = 0.4, allocating µprep = 0.24 and µpred = 0.32 under GDP composition; bounds derived from sensitive-data extrema are not fully private without accounting.Source: Private preprocessing in practice · p. 14

Membership-inference audit

  • The membership audit tested whether predictions exposed context membership using true-label confidence log-odds across 4,000 balanced draws of Maternal Health Risk rows.Each 507-row context draw came from 1,014 rows, with every row included in exactly 2,000 contexts.Source: Membership-inference audit · p. 14
  • The membership audit estimated record-AUC and balanced attack accuracy from fitted Gaussian member/non-member score distributions, using FPC-adjusted variances and confidence bounds.Record-AUC confidence bounds and balanced accuracy summarize empirical exposure but cannot establish differential privacy; the audit complements the mechanism-level guarantee.Source: Membership-inference audit · p. 15

Verification and auditing

  • Lean 4 verified the sensitivity bound and adaptive GDP composition; automated checks tested row-wise transformations, while empirical audits probed sensitivity and private-output distinguishability.Gradient-based searches sought adjacent context datasets with maximally different pre-noise summaries, then likelihood-ratio tests compared sampled privatised outputs; agreement with the expected trade-off curve was not a substitute for proof.Source: Verification and auditing · p. 15

Code availability

  • The repository provides source code, pretrained PrivTab weights, documentation, and instructions for reproducing the experiments.Source: Code availability · p. 15

Extended figures

PrivTab maps context and target rows to embeddings, then uses private cross-attention to compress sensitive context into summaries reused by the prediction module.

Extended figures

  • Context and target rows are first mapped to row embeddings by RowEmbed.Source: Figure E1 · p. 20
  • Data-independent learnable pseudo-tokens cross-attend to sensitive context rows through DP-MHCA, producing privatised summaries.Source: Figure E1 · p. 20
  • The public Prediction module reuses the summaries through cross-attention and token-wise multilayer perceptrons; target-side computation is post-processing.DP-MHCA is the only operation that transfers context information into the released summaries.Source: Figure E1 · p. 20

S1 Detailed Background

This section situates PrivTab as an amortised, in-context approach to tabular classification with differentially private context summarisation. It outlines the privacy and attention mechanisms behind the design and distinguishes it from non-private foundation models, private optimisation, and related approaches to other tasks or forms of privacy.

S1.2 Differential Privacy

  • Differential privacy limits how much an algorithm’s output distribution changes between adjacent datasets; post-processing preserves the guarantee, and this work uses substitute adjacency.The definition quantifies the bound over possible output events for adjacent datasets.Source: S1.2 Differential Privacy · p. 22

S1.3 Gaussian Differential Privacy

  • Gaussian differential privacy expresses guarantees through hypothesis-testing trade-offs, supports tight adaptive composition, and provides a Gaussian mechanism whose guarantee depends on noise and function sensitivity.The paper uses µ-GDP to report privacy guarantees.Source: S1.3 Gaussian Differential Privacy · pp. 22–23

S1.4 Bayesian Inference; S1.5 Stochastic Processes

S1.6 Amortised Bayesian Prediction

  • Amortised Bayesian prediction learns to map context data directly to posterior predictive distributions, minimizing expected target-label negative log-likelihood without performing inference from scratch for each task.The task contains observed context and target sets, not the latent parameter.Source: S1.6 Amortised Bayesian Prediction · pp. 24–25

S1.7 Neural Processes and Conditional Neural Processes

  • Conditional neural processes encode context pairs, aggregate them with a permutation-invariant operation, then combine the summary with each target input to predict; simple pooling can miss row interactions and target-specific relevance.PrivTab follows the deterministic conditional formulation, with its attention-based latent bottleneck described in the following subsection.Source: S1.7 Neural Processes and Conditional… · p. 25, S1.8 Transformer Neural Processes · p. 25

S1.8 Transformer Neural Processes

  • Transformer neural processes use attention to model context-row interactions and target-specific relevance; row permutations preserve predictions, but feature-order invariance is not guaranteed.Context rows are invariant under permutation, while permuting target rows permutes their corresponding predictions.Source: S1.8 Transformer Neural Processes · pp. 25–26

Transformer Neural Processes.

  • PrivTab adopts an LBANP-style fixed set of summary tokens and prior-data fitting (Müller et al., 2022), placing privacy at the context-to-summary attention interface before downstream prediction.The design draws on latent bottlenecks to reduce the cost of full attention while retaining target-dependent prediction.Source: Transformer Neural Processes · p. 27

S1.9 Related Work

  • PrivTab adds differential privacy to the in-context approach of TabPFN (Hollmann et al., 2023, 2025) and TabICL (Qu et al., 2025, 2026), privatizing a compact dataset representation once rather than using private stochastic optimization (Abadi et al., 2016; Ponomareva et al., 2023).The compact representation is reused by post-processing; the related-work passage also reports high membership-inference attack success for TabPFN v3 and TabICL v2.Source: S1.9 Related Work · p. 27
  • DPConvCNP (Räisä et al., 2024) studies private convolutional regression, while other work applies DP during meta-training rather than privatising an inference-time context set for amortised prediction.These approaches differ from PrivTab’s focus on heterogeneous tabular classification.Source: S1.9 Related Work · pp. 27–28
  • Recent DP in-context learning work covers private prompts, response aggregation, few-shot examples, nearest-neighbor retrieval, and prompt learning; DP-TabICL (Carey et al., 2024) turns private tabular records or statistics into LLM examples.The cited approaches include Gao et al. (2025), Wu et al. (2024), Tang et al. (2024), Koskela et al. (2025), and Duan et al. (2023).Source: S1.9 Related Work · p. 28
  • PrivTab is a standalone tabular model that builds privacy into context summarisation, requiring neither external LLM prompting nor synthetic-query generation to fit a new dataset.The compared DP-ICL methods address text prompts and LLM inference rather than tabular amortised classification, though they share the goal of controlling information released from sensitive context data.Source: S1.9 Related Work · p. 28
  • Local DP embedding methods privatise users’ representations before release, whereas PrivTab applies central DP to a shared context dataset for tabular classification.The cited transformer-text-embedding approach uses a variational information bottleneck and characterises privacy with Rényi divergence and Bayesian Differential Privacy (Zein and Henderson, 2026).Source: S1.9 Related Work · p. 28

S2 Noise-aware Amortised Inference

PrivTab’s noise-aware target is the distribution of target labels given public target inputs and the observed private summary. The amortised predictor is trained to approximate this target using an expected negative log-likelihood objective.

S2 Noise-aware Amortised Inference

  • The noise-aware posterior predictive conditions target labels on public target inputs and the observed private summary, rather than directly on the confidential context dataset.It accounts for uncertainty from both the latent parameter and the randomness of the fixed private mechanism.Source: S2 Noise-aware Amortised Inference · p. 28
  • For fixed mechanism parameters, PrivTab learns an amortised predictive distribution from the expected negative log-likelihood objective using the released DP output instead of the confidential context.The private prediction objective is the expected NLL defined for this setting.Source: S2 Noise-aware Amortised Inference · p. 28
  • At fixed target size and mechanism parameters, minimizing the conditional NLL is equivalent to minimizing expected KL divergence from the noise-aware target to the amortised predictor.The equivalence follows because the remaining objective term does not depend on the predictor parameters.Source: S2 Noise-aware Amortised Inference · pp. 28–29

S3 Differentially Private Transformer Neural Processes

This section describes PrivTab’s differentially private prediction architecture, which uses privacy-preserving attention to compress sensitive context rows into privatized summaries before predicting target queries through post-processing.

S3 Differentially Private Transformer Neural Processes

  • PrivTab embeds context and target rows, aggregates context tokens into privatized summaries with DP-MHCA, and lets target tokens attend to those summaries before prediction.The mechanism uses bounded per-row contributions followed by Gaussian perturbation; target-side computation requires no additional privacy protection.Source: S3 Differentially Private Transformer… · p. 30, S3.2 DP-MHCA · pp. 31–32

S3.1 Row Embedding Model

  • The row encoder is not exactly feature-order invariant; randomized synthetic-task pretraining encourages practical robustness, but provides no architectural guarantee.Row-wise processing supports privacy analysis and row-order symmetry, whereas the feature-specific MLP uses coordinate-specific weights.Source: S3.1 Row Embedding Model · p. 31

S3.2 DP-MHCA

  • With fresh Gaussian noise calibrated across Lmain DP-MHCA layers, the encoder satisfies µ-GDP under substitute adjacency, and its sensitivity bound does not grow with context size.Target-side computation is post-processing and adds no further privacy loss.Source: S3.2 DP-MHCA · pp. 33–34
  • When a fixed fraction of context rows aligns with a query and the remaining contribution stays bounded, directional signal grows linearly with informative rows without increasing the required DP noise scale.Aligned rows reinforce one another; the theorem formalizes when additional context can improve signal-to-noise ratio.Source: S3.2 DP-MHCA · p. 34

S3.3 Reference Implementation

  • The reference module uses 128 learned pseudo-tokens, three private cross-attention blocks, and two additional latent self-attention blocks, returning five summary states.Its guarantee assumes fixed weights, row-wise context embeddings, fresh independent Gaussian noise, and public or separately privatized dataset-level preprocessing; private releases should use a secure sampler.Source: S3.3 Reference Implementation · p. 34
  • The reference implementation uses 128 learned 256-dimensional latents, three DP cross-attention and three self-attention layers, followed by two latent self-attention layers.It returns five summary states for the target-side predictor.Source: S3.3 Reference Implementation · p. 36

S3.4 PrivTab Architecture

  • The architecture compresses the private context dataset into five fixed-size privatized summary states and classifies public target queries using non-private post-processing.Its three components are transformer building blocks, a Perceiver-style private encoder, and an output decoder.Source: S3.4 PrivTab Architecture · p. 36

S3.4.1 Transformer Building Blocks

  • The transformer blocks use pre-normalization, residual connections, standard multi-head attention, and SwiGLU feed-forward networks with a 2× hidden-dimension expansion.Layer normalization precedes each attention or feed-forward block, with the residual connection added afterwards.Source: S3.4.1 Transformer Building Blocks · pp. 36–37

S3.4.2 Private Perceiver Encoder

  • PrivTab’s Perceiver-style encoder uses DP-MHCA to transfer private context into summaries, then updates targets using those privatized summaries.A single-row change affects at most one context token; DP-MHCA is the only operation that transfers context information into released summaries, and later post-processing layers do not access the private context.Source: S3.4.2 Private Perceiver Encoder · pp. 37–38

S3.4.3 Output Decoder

  • The output decoder is a row-wise MLP with GELU activations that maps final target representations to predictive logits.Its hidden dimension is 512; for datasets with C classes, only the first C coordinates of the Cmax logits are used.Source: S3.4.3 Output Decoder · pp. 38–39

S4 PrivTab Pretraining

PrivTab is pretrained to predict from privatised summaries by learning the posterior predictive conditioned on those summaries. Training averages over simulated tasks, target sizes, and privacy levels, with curricula for both dataset scale and privacy noise.

S4.1 Noise-Aware Pretraining Objective

  • The noise-aware objective trains predictions to approximate the posterior predictive conditioned on the privatised summary, rather than ignoring privacy noise.With the private encoder fixed, the predictive network fits that distribution; joint training also changes the summary mechanism and its induced predictive family.Source: S4.1 Noise-Aware Pretraining Objective · p. 39

S4.2 Simulator, Curriculum, and Optimisation

  • Pretraining draws synthetic classification tasks from a mixed structural causal model, with labelled context and target sets supplying each task.The model is trained on target-label cross-entropy, equivalently negative log-likelihood, with label smoothing used as a training regulariser.Source: S4.2 Simulator, Curriculum, and… · pp. 39–41
  • A two-stage dataset-size curriculum moves from smaller to larger context sets, and separate small- and large-context variants are routed by context size at inference.The stages use context ranges [100, 2048] and [1024, 8192]; inference uses the small-context variant below 4096 rows and the large-context variant at or above 4096.Source: S4.2 Simulator, Curriculum, and… · p. 40
  • The final objective averages private negative log-likelihood across target sizes and privacy levels, minimizing divergence to the corresponding noise-aware posterior predictive.A curriculum first exposes the model to lower-noise tasks, then gradually introduces higher-noise settings.Source: S4.2 Simulator, Curriculum, and… · pp. 40–41

S5 Baseline Details

The supplement specifies how DP-Adam privacy is accounted for and how private-baseline hyperparameters are selected. Its protocol calibrates noise to the target guarantee and reserves part of the privacy budget for learning-rate selection.

S5.1 Accounting for DP Stochastic Optimisation under Substitute Adjacency

  • At each optimization step, DP Adam Poisson-samples a minibatch, clips each per-example gradient to norm C, adds noise to the clipped gradient sum, and applies Adam.Source: S5.1 Accounting for DP Stochastic… · p. 41
  • The accountant composes the Poisson-subsampled Gaussian mechanism’s privacy profile over training steps and converts it to the smallest valid non-asymptotic µ-GDP guarantee under substitute adjacency.It uses a numerical privacy-loss-distribution accountant rather than the asymptotic µ-GDP approximation (Bu et al., 2020b).Source: S5.1 Accounting for DP Stochastic… · p. 41
  • For each target privacy budget, the DP-Adam noise standard deviation is chosen by numerically inverting the accountant’s map.Source: S5.1 Accounting for DP Stochastic… · p. 42

S5.2 Hyperparameter Calibration and Private Selection

S6 Additional Experiment Details

The supplementary experiments document record-level membership audits, verification of the private mechanism, benchmark comparison conventions, and per-dataset results. They also clarify runtime measurement and the preprocessing and scoring conventions used in the aggregate comparisons.

S6.1 Membership-Inference Audit Results

  • Table S5 reports FPC-corrected aggregate and record-level membership-AUC estimates, while record-level curves provide threshold-specific exposure views.The table includes means and complementary 95th-percentile estimates; the record-level 95th percentile summarizes the most exposed 5% of rows.Source: Table S5 · p. 45, S6.1 Membership-Inference Audit Results · p. 45
  • PrivTab’s most vulnerable-record ROC curves remain closer to the perfect-privacy diagonal than those of non-private TabPFN v3 and TabICL v2, whose most vulnerable records are nearly perfectly distinguishable.Source: Figure S3 · p. 45
  • The reported largest lower record-wise confidence endpoint is pointwise, not a simultaneous 95% coverage guarantee across records.Source: S6.1 Membership-Inference Audit Results · p. 46

S6.2 Verification and Empirical Audits

  • Verification combines implementation checks, Lean 4 formalisation of the bounded-attention sensitivity bound and adaptive µ-GDP composition, and empirical audits.The formal objects’ correspondence to the running implementation remains subject to review; empirical agreement is not a substitute for the formal guarantee.Source: S6.2 Verification and Empirical Audits · pp. 46–47
  • Empirical audits test row-wise computation before the first private layer and search for adjacent token sets with large pre-noise summary differences; Fig. S4 reports trade-off curves across privacy levels.Agreement with the intended curve provides evidence against sensitivity-control or noise-calibration errors, but does not replace the guarantee.Source: S6.2 Verification and Empirical Audits · p. 47, Figure S4 · p. 47

S6.3.1 Preprocessing and Elo calibration

  • The comparison treats population-level z-score standardisation as idealised public preprocessing, while real-data evaluation estimates the statistics from the context set; Elo ratings are recalibrated without changing method rankings.Dataset-level AUC gaps average paired differences over ten context–target splits; the figures show gaps against DP-MLP and DP-LR.Source: S6.3.1 Preprocessing and Elo calibration · p. 48, Figure S5 · p. 48, Figure S6 · p. 49

S6.3.2 Derived comparison metrics

  • The derived comparisons report relative log-loss reductions and dataset-level win rates alongside AUC improvements, with positive AUC gaps favoring the compared method.Figure S7 plots AUC gaps against feature count across 33 datasets, privacy levels, and binary or multiclass tasks.Source: S6.3.2 Derived comparison metrics · p. 49, Figure S7 · p. 50

S6.4 Aggregate Benchmark Comparisons

S6.6 Private Preprocessing Pipeline Details

This supplement details PrivTab’s private preprocessing workflow and architectural ablations. It evaluates privately standardized models across five datasets, including at µ = 0.4.

S6.6 Private Preprocessing Pipeline Details

S6.6.1 Private moment mechanism

  • The private moment mechanism adds independent Gaussian noise to context sums and sums of squares, then uses noisy moments to standardize context and target inputs.Coordinates with very small or negative noisy variance, or constant clipping intervals, are set to zero; otherwise standardized inputs are clipped to [−10, 10].Source: S6.6.1 Private moment mechanism · p. 65

S6.6.2 LLM Bound-Inference Details

S6.7.1 Ablation: Noise-Aware Pretraining

  • Noise-aware pretraining ranks higher at every privacy level, while µ ∈ [0.16, 64] is the strongest overall privacy-range curriculum.Shifting the curriculum too far toward weaker or stronger privacy measurably degrades aggregate ranking.Source: S6.7.1 Ablation: Noise-Aware Pretraining · p. 70

S6.7.3 Ablation: Number of Private Layers

S6.7.4 Ablation: Number of DP-MHCA Heads

S6.7.5 Ablation: DP-MHCA Interaction Function

S6.7.6 Ablation: Context-Scale Normalization and Model Combination

  • The context-scale-aware combination achieves the best aggregate ranking, routing contexts below 4096 to the small model and others to the large model.The compared small- and large-context models use normalization.Source: S6.7.6 Ablation: Context-Scale… · p. 73
Loading 2610.10068v1…