Beyond Marginals: A Multi-Dimensional Evaluation Framework for Multi-Table Synthetic Data Generation

Evaluation gaps

• Existing evaluation often tests columns independently, missing preserved inter-column correlations and conditional distributions.Related gaps include unassessed cross-table integrity, fragmented utility and privacy metrics, absent production-readiness checks, and ad hoc generator comparisons.Source: §1 · pp. 1–2

SynEval

• SynEval formalizes six dimensions: per-column fidelity, multivariate structure, cross-table integrity, ML utility, privacy, and edge-case robustness.The framework targets multi-table synthetic data.Source: §1 · p. 2

Novel metrics

• Novel checks assess conditional distributions P(Y | X ∈ bin), cross-table cardinality matching, and edge cases including nulls, rare categories, zero-inflation, tails, and mode collapse.The edge-case battery also tests percentile accuracy.Source: §1 · p. 2

Scoring and implementation

• A unified weighted score supports reproducible generator comparisons with per-table and per-dimension drill-down.The implementation is generator-agnostic, with automatic schema inference and a visualization suite.Source: §1 · p. 2

2 Related Work

• Related work spans single-table evaluation, multi-table generators and evaluation, and privacy-specific assessment.The paper organizes its review into these four areas.Source: §2 · p. 2

2.1 Single-Table Synthetic Data Evaluation

• Single-table tools assess column-level quality or combine fidelity, utility, and privacy, but do not assess relational structure preservation.SDMetrics offers marginal metrics, while TableEvaluator and TabSynDex treat tables independently; Dankar et al. [2022] and Wang et al. [2023] remain single-table approaches.Source: §2.1 · p. 2

2.2 Multi-Table Synthetic Data Generators

• HMA, RC-TGAN, and ClavaDDPM model relational data through parent-to-child sampling, parent features, or diffusion with Gaussian Mixture intermediaries.Their differing evaluation metrics motivate a standardized framework.Source: §2.2 · p. 2

2.3 Multi-Table Evaluation

• Existing multi-table evaluation remains sparse: SDMetrics includes cardinality shape and parent-child detection, but omits conditional subgroup distributions and edge cases.Its single aggregate score also masks per-table and per-dimension diagnostics; SyntheRela focuses on detection-based fidelity and utility, while the paper identifies no framework jointly assessing all six proposed dimensions.Source: §2.3 · p. 3

2.4 Privacy Evaluation for Generative Models

• Privacy metrics such as DCR, NNDR, and membership inference are established individually; the paper integrates them into a composite privacy dimension.The dimension is weighted alongside fidelity, utility, and structural metrics.Source: §2.4 · p. 3

3.1 Setting and Notation

• The setting denotes a real relational database as Dreal with K tables and a schema S specifying database structure.The notation introduces table keys, relationships, and column-type annotations.Source: §3.1 · p. 3
• PK(Tk) denotes each table’s primary key, FK(Tk) its foreign-key references, and Rel the set of parent–child relationships.Non-key columns are partitioned into continuous and categorical types.Source: §3.1 · p. 4
• A multi-table generator maps the real database and schema to a synthetic database with the same schema.The generated database is denoted Dsynth.Source: §3.1 · p. 4

3.2 The Quality Function

• The quality function Q(Dreal, Dsynth, S) maps a real and synthetic database pair and their schema to a score in [0, 1].The formulation specifies comprehensiveness, decomposability, and generator-agnosticism as design properties.Source: §3.2 · p. 4
• Q is comprehensive: it covers statistical fidelity, structural integrity, downstream utility, privacy, and production robustness rather than marginals alone.The six dimensions include per-column fidelity, multivariate structure, cross-table integrity, ML utility, privacy, and edge-case robustness.Source: §3.2 · p. 4
• Q decomposes into per-table, per-dimension scores using configurable dimension weights and row-count weighting across tables.This weighting lets large transaction tables contribute more to the global score than small lookup tables.Source: §3.2 · p. 4
• Q is generator-agnostic: it evaluates output CSVs without coupling to generator internals.The pipeline can therefore compare generators through their outputs.Source: §3.2 · p. 4

3.3 Failure Mode Taxonomy

• The failure-mode taxonomy identifies nine failure modes, including several that standard per-column tests cannot detect.Modes 2–4 and 8–9 are specifically described as invisible to those tests; Table 1 presents the taxonomy.Source: §3.3 · p. 4, Table 1 · p. 5

4.1 Architecture Overview

• SynEval loads real and synthetic CSV folders, resolves schema, evaluates five dimensions per table plus global cross-table integrity, then aggregates scores with configurable weights.The architecture is presented as a five-stage pipeline.Source: §4.1 · p. 5

4.2 Dimension 1: Per-Column Fidelity

• Per-column fidelity compares continuous and categorical distributions with complementary metrics, adaptive thresholds, and a table-level pass-rate criterion.The dimension passes when the aggregate pass rate exceeds 70%; Benjamini–Hochberg correction at q = 0.05 upgrades nonsignificant KS or chi-square tests to pass.Source: §4.2 · p. 6

4.3 Dimension 2: Multivariate Structure

• Multivariate checks compare real and synthetic correlations, mutual information, and categorical association to assess inter-column dependencies beyond marginal distributions.Correlation differences use Pearson and Spearman; normalized mutual information and Cramér’s V use thresholds of 0.15.Source: §4.3 · p. 6
• Conditional checks compare distributions of continuous Y within categorical-X groups or four quantile bins of continuous X, capturing shape differences that correlation can miss.A generator could match the overall salary distribution and global salary–department correlation while producing identical salary distributions across departments.Source: §4.3 · p. 7

4.4 Dimension 3: Cross-Table Integrity

• Cross-table integrity assesses orphaned foreign keys, child-count distributions, parent-level correlations, and synthetic values outside real-data bounds.The composite score weights FK integrity at 0.35, bounds at 0.15, cross-table correlation at 0.15, cardinality at 0.15, and conditional checks at 0.20; the pass threshold is 0.70.Source: §4.4 · p. 7

4.5 Dimension 4: ML Utility

• ML utility uses TSTR efficacy ratio, training on synthetic data and testing on held-out real data, with classification F1 or regression R2.The dimension passes if ρ ≥ 0.90; the target is user-specified when configured, otherwise selected by column type and class count.Source: §4.5 · p. 7

4.6 Dimension 5: Privacy

• Privacy checks use Gower distances, closest-record distance, nearest-neighbor distance ratios, and exact-match detection to assess memorization risk.To avoid penalizing matches to legitimate real-data duplicates, exact matches are also counted only against unique real records, with raw and adjusted scores reported.Source: §4.6 · p. 8

4.7 Dimension 6: Edge-Case Robustness

• Edge-case robustness averages pass rates across checks for null patterns, rare categories, tails, zero inflation, mode collapse, value bounds, and percentile accuracy.The dimension passes if its mean sub-metric pass rate is at least 0.70.Source: §4.7 · p. 8

5.1 Benchmark Datasets

• The benchmarks are a purpose-built six-table E-Commerce database with 13 cross-table edge cases and an 11-table F1 database with deep and multi-parent relationships.The F1 dataset has approximately 534,000 rows and is 20× larger than E-Commerce, with 3× more relationships and row counts ranging from 73 to 420,369.Source: §5.1 · pp. 8–10

5.2 Generators Under Evaluation

• Three generators represent diffusion-based, GAN-based, and hierarchical conditional-GAN approaches to multi-table synthesis.They are ClavaDDPM, RC-TGAN, and RSDG, respectively; all used default hyperparameters without tuning, to demonstrate diagnostic workflow rather than provide a definitive ranking.Source: §5.2 · pp. 10–11

5.3 Evaluation Protocol

• Each generator’s output was evaluated with all six SynEval dimensions and default weights, using JSON schemas for keys, relationships, and column types.Source: §5.3 · p. 11

6.1 Overall Results

• ClavaDDPM scores 87.0% on F1 and 75.9% on E-Commerce, the highest overall scores; RSDG reaches 76.0% on E-Commerce.Table 3 reports the cross-dataset, cross-generator comparison.Source: Table 3 · p. 11, §6.1 · p. 11

6.2 Radar Comparison

• ClavaDDPM leads on Multivariate and Cross-Table structural dimensions, while GAN-based generators show stronger ML Utility.Figure 4 compares six dimension scores across three generators for each dataset; the dashed circle marks the 0.70 pass threshold.Source: §6.2 · p. 11, Figure 4 · p. 12

6.3 ML Utility: A Cross-Dataset Challenge

• ML Utility varies most across generators and datasets: ClavaDDPM leads structurally but has the lowest F1 utility, while RSDG leads utility and overall score on E-Commerce.Strong distributional preservation does not guarantee predictive utility.Source: §6.3 · p. 12
• Heuristic target-column selection can favor easier or harder prediction tasks, making ML Utility scores less stable and interpretable.Domain-informed target selection would yield more stable and interpretable scores.Source: §6.3 · p. 12

6.4 Privacy: Dataset-Dependent

• Privacy scores depend on dataset size: all generators score 78–82% on F1 but 47–53% on E-Commerce, whose small tables enable near-memorization.The passage attributes F1’s stronger privacy scores to its larger table sizes.Source: §6.4 · p. 12

6.5 Cross-Table Integrity

• Cross-Table Integrity is strongest overall; score differences reflect cross-table correlations and conditional preservation rather than foreign-key violations.Hierarchical parent-to-child sampling structurally prevents orphaned foreign keys.Source: §6.5 · pp. 12–13

7.1 Multi-Table Synthesis Remains a Hard Problem

• No generator exceeds 80% on both datasets: ClavaDDPM scores 87.0% on F1 and 75.9% on E-Commerce, while RSDG reaches 76.0% on E-Commerce.The authors describe preserving cross-table correlations as an open challenge.Source: §7.1 · p. 13

7.2 The Fidelity–Privacy Tension

• Privacy scores are 78–82% on the 534k-row F1 dataset but 47–53% on the 26k-row E-Commerce dataset; small tables with few distinct values make memorization nearly inevitable for current generators.The authors suggest interpreting privacy metrics relative to dataset size.Source: §7.2 · p. 13

7.3 ML Utility Validates Multi-Dimensional Evaluation

• All generators achieve 64–74% marginal fidelity but vary from 27–80% in ML Utility, so marginal-only assessment can miss degraded downstream ML utility.The authors present this disconnect as validation of SynEval’s multidimensional diagnostic approach.Source: §7.3 · p. 13

7.4 Experimental Scope

• The study tests three generators with default hyperparameters on two datasets; ML Utility also depends on target selection and one model family, XGBoost.Tuned configurations may yield better results, and different models or targets may change rankings.Source: §7.4 · p. 13

8 Conclusion and Future Work

• SynEval provides a decomposable quality score across per-column fidelity, multivariate structure, cross-table integrity, ML utility, privacy, and edge-case robustness.The authors present the framework as addressing five evaluation gaps, including the absence of unified scoring.Source: §8 · p. 13
• Dimension-level trade-offs, including structural fidelity versus ML utility, are surfaced, but generator-specific conclusions require caution.Hyperparameter tuning, architecture-specific preprocessing, and domain-informed target selection may substantially change individual scores.Source: §8 · p. 13
• Three capabilities are identified as novel: conditional distribution checks, cross-table cardinality matching, and edge-case robustness testing.The checks target failures missed by marginal and correlation tests, cardinality collapse, and cases such as null patterns, rare categories, zero-inflation, tail behavior, and mode collapse.Source: §8 · p. 14

8.1 Framework Limitations

• The framework does not evaluate temporal patterns, seasonality, or sequential dependencies.Future work proposes a temporal dimension covering seasonal decomposition, autocorrelation matching, and event-sequence fidelity.Source: §8.1 · p. 14, §8.2 · p. 14
• DCR and NNDR measure empirical distance but do not provide formal (ε, δ)-DP guarantees; formal guarantees require access to the generation mechanism, which conflicts with generator-agnostic design.Source: §8.1 · p. 14
• Default dimension weights reflect the authors’ assessment of relative importance rather than empirical derivation.Source: §8.1 · p. 14
• All experimental generators used default hyperparameters, and systematic training-time tuning may change the reported rankings and absolute scores.Source: §8.1 · p. 14

8.2 Future Work

• Future work proposes broader generator and dataset benchmarking, SynEval-guided tuning, evaluation of DP-enabled generators, and membership-inference attacks for privacy assessment.The proposed benchmarking includes additional generators and datasets with varying schema complexity; the privacy extensions would examine formal privacy trade-offs and subtler leakage.Source: §8.2 · p. 14

• The reference includes metric definitions and default thresholds.Source: A Complete Metric Summary · p. 16
• The reference also lists pass/fail criteria for the metrics.Source: A Complete Metric Summary · p. 16
• Table 4 is the complete metric reference for SynEval.Source: Table 4 · p. 16

B.1 Schema Auto-Inference

• Without a schema YAML, SynEval classifies object/category dtypes and numerics with ≤20 unique values as categorical, and remaining numerics as continuous.Columns ending in _id are excluded as keys; a primary key is inferred from the first fully unique column with an _id suffix.Source: §B.1 · p. 16

B.2 Per-Column Fidelity Details

• The adaptive KS critical value is clamped to [0.05, 0.30].Source: §B.2 · p. 17
• For JSD, real and synthetic values use 50 quantile-based bins with Laplace smoothing (ϵ = 10^-10).Quantile-based bins adapt to skewed distributions, unlike equal-width bins that may waste resolution on empty ranges.Source: §B.2 · p. 17
• For categorical chi-square tests, the p-value threshold is adjusted by α/max(1, |V|/5) to control family-wise error for high-cardinality categories.Source: §B.2 · p. 17

B.3 Multivariate Structure Details

• Multivariate evaluation discretizes continuous columns into 10 quantile bins for mutual information, excludes pairs with both NMI values below 0.01, and limits contingency tests to the top 15 categorical pairs.A tested pair passes when |Vreal − Vsynth| < 0.15 and chi-square p ≥ 0.05; continuous scoring uses σm = max(0, 1 − ∆max/(2τm)).Source: §B.3 · p. 17

• Rare-category preservation passes when at least 50% of categories below 1% frequency appear in the synthetic data.Source: §B.4 · p. 17
• Tail behavior passes when relative error at p5 and p95 is below 25%.Outlier rate is also tracked as P(|Xc − µc| > 3σc).Source: §B.4 · p. 17
• Mode collapse is flagged when the synthetic-to-real unique-value ratio is below 0.3.Source: §B.4 · p. 18
• Value bounds fail if any synthetic value falls outside the real data’s minimum–maximum range; percentile accuracy passes when the maximum IQR-normalized error across p5, p25, p50, p75, and p95 is ≤0.20.The edge-case pass rate aggregates pass rates across all seven sub-metrics.Source: §B.4 · p. 18