PaperlayerView exact revision on arXiv

Source-linked AI summary

Differentially Private Paired Table-Image Multimodal Synthesis

Kai Chen, Josephine Lamp, Somesh Jha, Tianhao Wang

arXiv:2609.00708v1cs.CRcs.AIcs.CV

TL;DR

Private paired table-image synthesis must preserve multivariate tabular structure, image quality, and cross-modal correspondence despite modality-specific privacy challenges. DP-TabImage combines a private tabular PGM, a table-conditioned DP-SGD diffusion model, and private paired prototypes for warm-up. Across three datasets, it achieves a stronger balance across these objectives, while visual and aligned warm-up signals improve different aspects of utility.

  • Problem

    Private synthesis of paired images and multivariate tabular records must preserve both modality marginals and their cross-modal dependence, which existing modality-specific approaches do not directly address.

  • Method

    DP-TabImage factorizes p(x,y) as pT(y)pI(x | y), using an AIM-based private PGM for tables, a table-conditioned DP-SGD diffusion model for images, and private table-image prototypes for warm-up.

  • Results

    Across three real-world datasets, DP-TabImage achieves a better balance among tabular fidelity, image fidelity, and cross-modal alignment than independent, joint, and reverse-factorized alternatives.

  • Takeaways & Limitations

    Visual prototypes primarily improve image quality, whereas aligned table-image prototypes are critical for preserving cross-modal correspondence.

  • Takeaways & Limitations

    The method uses coarse attribute-level prototypes and a low-resolution setting, leaving richer prototypes and higher-resolution synthesis as future challenges.

Abstract

from arXiv · show

Differentially private (DP) synthesis has been extensively studied for tabular and image data separately, yet many real-world datasets contain images paired with multivariate tabular records. Synthesizing such data is particularly challenging under DP, as the two modalities favor different private learning mechanisms while their dependence must also be preserved. To address this challenge, we propose DP-TabImage, a modality-specialized framework for private paired synthesis. DP-TabImage instantiates the factorization $p(x,y)=p_T(y)p_I(x\;|\;y)$ using a private Probabilistic Graphical Model for the multivariate table distribution and a table-conditioned diffusion model trained with DP-SGD for the conditional image distribution. To facilitate conditional learning under clipped and noisy gradients, we further pretrain the model on private table-image prototypes, pairing privately constructed attribute-conditioned images with tabular vectors derived from the already private tabular model at no additional privacy cost. Experiments on three real-world datasets show that DP-TabImage achieves a strong balance among tabular fidelity, image fidelity, and cross-modal alignment. Our analysis further reveals that visual warm-up primarily improves marginal image fidelity, whereas aligned table-image warm-up is critical for improving cross-modal correspondence. Our source code is available in the GitHub repository, https://github.com/KaiChen9909/TabImage_Syn.

1 Introduction

Paired table-image synthesis must preserve both modality-specific fidelity and dependencies between multivariate records and images under differential privacy. DP-TabImage addresses this with modality-specialized synthesis and private cross-modal initialization.

  • Motivation: Paired datasets require synthetic releases to preserve tabular fidelity, image fidelity, and cross-modal dependencies for downstream analysis.Randomly pairing independently synthesized modalities approximates pI(x)pT(y) and discards dependencies in p(x,y).
  • Challenge: Existing private methods favor marginal-based tabular synthesis or DP-SGD-trained image generation, making a single private joint model difficult to optimize.The model must learn multivariate statistics, visual structure, and alignment from clipped and noisy gradients.
  • Approach: DP-TabImage factorizes synthesis as p(x,y) = pT(y)pI(x | y), using an AIM-based private PGM for tables and a table-conditioned DP-SGD diffusion model for images.The factorization allows each modality to use a mechanism suited to its structure while conditional generation preserves dependence.
  • Approach: Private table-image prototypes expose the conditional image model to coarse associations before fine-tuning without additional privacy cost.Their tabular summaries come from the already-private PGM, while image prototypes are attribute-conditioned aggregates.
  • Findings: Across three real-world datasets, DP-TabImage balances tabular fidelity, image fidelity, and cross-modal alignment.Image-only prototypes improve image fidelity but provide little correspondence improvement, whereas table-image prototypes strengthen alignment.

2 Preliminary

The preliminary section defines differential privacy, its RDP accounting tools, DP-SGD, and the paired table-image synthesis problem. The target release must reproduce both marginal modality characteristics and cross-modal correspondence under an (ε,δ)-DP guarantee.

  • Differential Privacy: Differential privacy requires an algorithm's output distributions to remain controlled when neighboring datasets differ by one sample.The paper defines neighboring datasets using an add-or-remove-one-sample relation.
  • Differential Privacy: Rényi differential privacy provides a composition-oriented privacy accounting framework that can be converted into an (ε,δ)-DP guarantee.The paper states the RDP-to-DP conversion through its theorem.
  • DP-SGD: DP-SGD clips per-example gradients and adds Gaussian noise during optimization to control sensitivity under privacy.The update uses the loss gradient, clipping function, learning rate, and Gaussian noise level.
  • Problem Formulation: A table-image dataset contains paired fixed-resolution images and fixed-dimensional tabular records, along with their image and tabular marginals.The dataset is represented as N paired samples D = ((x1,y1),...,(xN,yN)).
  • Problem Formulation: DP table-image synthesis generates a synthetic paired dataset that mirrors the sensitive dataset under an (ε,δ)-DP guarantee.The task requires fidelity to both unimodal marginals and semantic correspondence between images and records.

3 Related Work

Related work covers private tabular synthesis, private image synthesis, and non-private paired generation, but these lines do not directly provide private synthesis of multivariate records with corresponding images. DP-TabImage is positioned to combine modality-specific mechanisms while retaining paired dependence.

  • DP Table Synthesis: AIM and related marginal-based methods privately estimate low-order statistics and synthesize tabular records with strong distributional fidelity.AIM is reported as strongest in most common tabular synthesis situations and motivates its use in DP-TabImage.
  • DP Image Synthesis: Feature-based private synthesis releases privatized statistics before generator optimization, avoiding additional privacy cost but generally yielding lower tabular or visual fidelity.The passage contrasts DP-MERF and DP-NTK with leading marginal-based and diffusion approaches.
  • Comparison: Table 1 compares DP-TabImage with adjacent synthesis settings, including unconditional and conditional DP image synthesis.In the conditional setting, c denotes a label or selected control variable.
  • DP Image Synthesis: Private image methods include unconditional and class-conditional generation, often conditioning on a single label or low-dimensional control variable.These methods preserve simpler image-control associations but do not separately model a complex multivariate tabular distribution.
  • Non-DP Paired Synthesis: Non-private paired-generation methods model images and tabular records jointly but lack end-to-end DP guarantees and are not directly amenable to private training.Their multiple data-dependent components would require privatizing every sensitive-data access during adaptation.

4 Method

DP-TabImage uses a table-first factorization to assign tabular and image synthesis to different private mechanisms while preserving dependence through conditioning. The base workflow privately models tables with a PGM, then generates an image for each synthetic tabular record.

  • Modality-Specialized Factorized Synthesis: The method decomposes the paired distribution as p(x,y) = pT(y)pI(x | y), with tables generated first and images generated conditionally.This direction makes fixed-dimensional tabular records suitable as conditioning variables for the image generator.
  • Modality-Specialized Factorized Synthesis: The factorization lets tabular and conditional image distributions use different private synthesis paradigms instead of one DP-SGD-trained joint model.This design separates modality-specific learning requirements while retaining conditional dependence.
  • Base Pipeline: DP-TabImage instantiates the table distribution with AIM's private PGM and the conditional image distribution with a table-conditioned DPDM.The base pipeline first samples complete tabular records and then generates one conditioned image per record.
  • Base Pipeline: Algorithm 1 trains the private PGM, initializes the conditional diffusion denoiser, and performs iterative private image-model training before producing synthetic data.The listed workflow includes training steps, sampling rate, learning rate, and noise level as algorithm inputs.

11 Dsyn

DP-TabImage constructs private paired table-image data by modeling tabular records with a PGM, conditioning diffusion on complete records, and warm-starting with aligned private prototypes.

  • Private Conditional Image Modeling: The conditional image model injects the complete tabular record into the U-Net through a tabular encoder and cross-attention modules.It is trained on private pairs with DP-SGD and predicts the denoising target conditioned on all tabular variables.
  • Factorized Sampling: The factorized sampler first generates tabular records with a private PGM, then uses them as conditions for diffusion-based image generation.This separates modality-specific private synthesis while retaining dependence through conditional generation.
  • Prototype Warm-up: Private table-image prototypes warm up the conditional generator, reducing the burden of learning visual structure and cross-modal associations from noisy gradients alone.The prototype dataset contains aligned visual and multivariate tabular information.
  • Mean Image Construction: Mean images are constructed by subsampling paired records, grouping images by categorical values or discretized numerical bins, and aggregating the corresponding images.The construction uses noisy image counts and image sums, with image-sum sensitivity denoted by ∆x.
  • Mean Tabular Condition Construction: Soft tabular prototype vectors fix one target attribute and fill the remaining attributes with conditional marginals from the private PGM.Categorical targets use one-hot representations, while numerical targets use discretized-bin midpoints.
  • End-to-End Procedure: The complete algorithm builds the PGM and prototypes, pretrains the conditional denoiser, then performs DP-SGD fine-tuning and paired generation.Prototype-derived quantities and pretraining are based on already private outputs.

10 Apply data augmentation to ¯D

DP-TabImage combines private table modeling, prototype construction, and DP-SGD image fine-tuning, with privacy accounting through RDP composition.

  • Privacy Cost Analysis: The privacy analysis accounts for AIM-based table modeling, private prototype construction, and DP-SGD fine-tuning of the conditional image generator.Operations derived from the private PGM or prototypes, including pretraining and synthetic generation, incur no additional privacy cost.
  • Prototype Privacy: The prototype mechanism releases noisy count vectors and noisy image-sum vectors for each tabular attribute.The image-sum noise scale is expressed using σx∆x, while count vectors use Gaussian noise with standard deviation σm.
  • Utility Evaluation: Table 2 evaluates tabular utility with 3-way TVD, image utility with FID, and cross-modal correlation with downstream AUC or MAE.Downstream MAE for numerical attributes appears only in the Chest dataset.
  • RDP Composition: For every α > 1, the end-to-end method satisfies (α,γtotal(α))-RDP by composing AIM, prototype construction, and sampled Gaussian mechanisms.The sampled Gaussian terms correspond to the DP-SGD steps.
  • Differential Privacy Guarantee: For every 0 < δ < 1, the composed RDP guarantee yields an (ε,δ)-DP guarantee.The detailed proofs are provided in Section E.

5 Experiments

Experiments evaluate DP-TabImage across three paired datasets and three utility dimensions. The method provides strong tabular fidelity, image fidelity, and cross-modal alignment, while warm-up and privacy allocation produce distinct utility trade-offs.

  • Experimental Setup: Experiments use DeepFashion, CelebA, and Chest to assess tabular fidelity, image fidelity, and cross-modal alignment.Tabular fidelity uses 3-way marginal TVD, image fidelity uses FID, and alignment uses downstream prediction metrics.
  • End-to-End Synthesis Comparison: DP-TabImage combines an AIM-based private PGM for tables with a table-conditioned private diffusion model for images.Its table-first factorization is p(x,y) = pT(y)pI(x | y), with aligned table-image warm-up prototypes.
  • End-to-End Synthesis Comparison: DP-TabImage provides superior tabular utility, attributed to decoupling tabular generation and using AIM to capture tabular information through marginals.DP-SGD-based alternatives show limitations in accurately modeling tabular distributions under clipping and noise.
  • End-to-End Synthesis Comparison: DP-TabImage attains the lowest FID in five of six dataset-budget settings and remains comparable to the best Chest baseline at ε = 10.The factorized design leaves more privacy budget for image generation, while warm-up improves initialization and lowers FID.
  • End-to-End Synthesis Comparison: DP-TabImage has a significant AUC advantage over baselines, while algorithms show similar MAE; with ε = 10, its performance is close to ground truth.Alternative joint or reverse-factorization methods can be worse than DP-TabImage-Base on CelebA and Chest under ε = 10.
  • Warm-up Module Analysis: Image-only warm-up improves marginal image fidelity, whereas aligned table-image warm-up consistently improves cross-modal metrics and yields a better paired-data balance.All warm-up variants reduce training loss earlier; DP-TabImage improves AUC rapidly but can have slower or higher FID than image-only variants.
  • Privacy Allocation Ablation: FID is more sensitive to privacy allocation than tabular fidelity or alignment: reducing the tabular budget improves FID by 4.2%, while TVD and AUC vary by at most 0.3% and 0.2%.FID increases monotonically with λT, whereas TVD and prediction AUC show no comparable monotonic trend across evaluated allocations.

6 Discussion and Limitations

DP-TabImage trades some image quality for better cross-modal alignment, while its low-resolution, attribute-pruned setting limits how broadly the conclusions apply. The study also excludes auxiliary public data and leaves richer prototype construction for future work.

  • Trade-offs and future work: DP-TabImage sacrifices some image quality to obtain better cross-modal alignment.The authors note that richer attribute-level prototypes might improve alignment without reducing image fidelity.
  • Scope boundaries: Low-resolution images obscure fine-grained visual information and require excluding some attributes, limiting extension to high-resolution or rare-condition settings.The authors also identify extreme class imbalance as a reason for excluding attributes.
  • Scope boundaries: The study excludes auxiliary public data, although public data and pretrained representations may improve visual fidelity under different availability assumptions.

7 Conclusion

The paper concludes that DP-TabImage balances tabular fidelity, image fidelity, and cross-modal alignment across three datasets. It emphasizes that visual prototypes mainly improve image quality, whereas aligned table-image prototypes are critical for preserving correspondence.

  • Conclusion: DP-TabImage combines an AIM-based private PGM, a table-conditioned diffusion model, and private paired-data warm-up.
  • Conclusion: Across three datasets, DP-TabImage achieves a better balance among tabular fidelity, image fidelity, and cross-modal alignment than independent, joint, and reverse-factorized alternatives.
  • Conclusion: Visual prototypes primarily improve image quality, whereas aligned table-image prototypes are critical for preserving cross-modal correspondence.
  • Study scope: The study uses publicly available datasets and does not involve human subjects, private user data, or real-world vulnerability analysis.
  • Reproducibility: The authors provide code, dataset, and execution instructions in an open-source repository.
  • Data preparation: Images are center-cropped and downsampled because current DP generative algorithms struggle with high-dimensional data.
  • Data preparation: Fine-detail and extremely imbalanced tabular attributes are removed after resizing reduces visual information.
  • Data preparation: Datasets are randomly split into training, testing, and validation sets using a 7:1:2 ratio.

B Evaluation Metrics Details

The evaluation combines tabular fidelity, image fidelity, downstream prediction, and representation-level cross-modal alignment metrics. It also specifies preprocessing, dataset statistics, privacy settings, and prototype construction procedures.

  • Evaluation metrics: Marginal TVD measures error between discretized numerical-attribute distributions using 100 bins per public value domain.
  • Evaluation metrics: FID follows the implementations used by DP-FETA and DPDM, while downstream prediction uses a CNN backbone with modular multi-head outputs.
  • Evaluation metrics: The downstream model jointly minimizes classification and regression errors through a composite multi-task loss.
  • Experimental setup: The experiments report dataset statistics, hyperparameters, privacy histories, TVD, downstream F1, accuracy, MSE, and ESG comparisons.
  • Evaluation metrics: Exponential moving averages and validation-based early stopping support final evaluation and overfitting control.
  • Cross-modal alignment: ESG trains fixed image and tabular encoders on real pairs and measures cosine similarity in a shared normalized embedding space.A multi-positive contrastive objective handles repeated attribute vectors.
  • Cross-modal alignment: A larger ESG indicates better preservation of cross-modal correspondence, and reported ESG is normalized by the value on real test pairs.
  • Prototype construction: Prototype construction downscales images, adds Gaussian noise, upsamples noisy means, and discards attribute-value groups with noisy counts below 100.

D.1 Extra Main Exprimental Results

Supplementary experiments show that DP-TabImage preserves low-order marginals, downstream cross-modal performance, and embedding-level alignment, while non-private comparisons favor the table-first factorization. The authors caution that architecture choices and low-resolution preprocessing constrain interpretation.

  • D.1 Extra Main Exprimental Results: DP-TabImage, DP-TabImage-Base, and DP-Indep retain identical low-order TVD errors across ε = 1 and ε = 10.The authors attribute this to AIM already receiving enough budget to recover low-order marginals.
  • D.1 Extra Main Exprimental Results: DP-TabImage achieves the strongest downstream performance, including 0.43 F1 and 0.78 Acc at ε = 1 on CelebA.At ε = 10 on CelebA, it reaches 0.52 F1 and 0.82 Acc.
  • D.1 Extra Main Exprimental Results: DP-TabImage reaches the best ESG among baselines and variants, attaining 0.132 at ε = 1 on CelebA.Figure 8 shows the same ordering across DeepFashion, Chest, and CelebA.
  • D.1 Extra Main Exprimental Results: Aligned tabular summaries, rather than image prototypes alone, account for the stronger cross-modal alignment in the variant comparison.
  • D.2 Extra Attribute-Level Results: Across most attributes in Figure 9, DP-TabImage generally achieves better attribute-level cross-modal performance than the other baselines.Attributes are ordered by ground-truth performance, with ε = 1 in the upper row and ε = 10 in the lower row.
  • D.3 Non-DP Results: Without DP, the table-first factorization reaches 0.014 3-way TVD versus 0.292 for the joint design and 0.325 for the reverse-factorized design.
  • D.3 Non-DP Results: ImageTab obtains the lowest FID at 6.89, while TabImage trades some image fidelity for stronger tabular and cross-modal quality.
  • D.3 Non-DP Results: The non-private comparison uses one architecture per paradigm, so stronger joint or reverse-factorized models might narrow the observed gap.
Loading 2609.00708v1…