PaperlayerView exact revision on arXiv

Source-linked AI summary

Position: Privacy Is a Claim, Not a Property of Synthetic Data

Jiachen Zhao, Antonia Januszewicz, Taeho Jung

arXiv:2609.01273v1cs.LGcs.CR

TL;DR

Synthetic data is often treated as privacy-preserving based on its generation process rather than verifiable evidence about residual inference risk. This position paper combines an analysis of recent ML publications with a framework for explicit privacy claims, finding that verification remains rare and arguing for scoped, testable, contestable assertions.

  • Problem

    Synthetic data is increasingly treated as a default response to privacy concerns, despite privacy claims often lacking formalization, threat models, or verifiable guarantees.

  • Method

    The paper analyzes privacy-sensitive uses of synthetic data in recent ML publications and proposes a Minimum Privacy Claim Standard for distinguishing risk-reduction heuristics from evidence supporting privacy claims.

  • Results

    Verification remains rare in privacy-sensitive uses of synthetic data, while generation alone cannot establish privacy and surface audits may miss statistically detectable training-data traces.

  • Takeaways & Limitations

    Privacy should be treated as an explicit, verifiable, evidence-based claim about residual risk rather than an implicit property of synthetic data.

  • Takeaways & Limitations

    Synthetic data remains useful as a risk-mitigating heuristic, but that operational role should not be interpreted as evidence supporting a privacy claim.

Abstract

from arXiv · show

Synthetic data has become a common component of machine learning research. While widely adopted, its use in privacy-sensitive contexts has quietly shifted from a claim of residual inference risk under stated assumptions to an appearance-based property inferred from data generation itself. In this position paper, we argue that this shift reflects an implicit change in community standards for what counts as sufficient privacy evidence, rather than a misunderstanding of well-established privacy principles. Drawing on an empirical analysis of recent publications across major ML venues, we show that synthetic data is frequently used in privacy-sensitive settings without explicit articulation of threat models, inference risks, or falsifiable privacy claims. As a result, privacy assurance often remains implicit, difficult to verify, and unevenly distributed, with heightened exposure for rare and minority records. We argue for treating privacy as an explicit, evidence-based scientific claim and recommend that ML venues adopt norms requiring privacy-relevant assertions to be clearly scoped, testable, and contestable.

1. Introduction

Synthetic data is increasingly treated as privacy-preserving based on how it is generated rather than on explicit evidence about residual inference risk. The paper argues for restoring privacy as a verifiable claim and develops an empirical audit and a minimum privacy claim standard.

  • Synthetic data is often described as “privacy-preserving” or “safe by construction” without formalizing the privacy risks or guarantees involved.This appearance-based framing shifts privacy from an empirical inference-risk claim toward an inferred property of generation.
  • Surface properties such as non-identity, novelty, and absent direct replicas can overlook training dependence and leave inference risk intact.The paper also argues that privacy risk may be redistributed toward rare or atypical records.
  • The paper re-grounds privacy in explicit, verifiable, evidence-based claims rather than treating synthetic generation itself as proof of privacy.Synthetic generation may reduce exposure and enable access, but the paper states that it cannot establish privacy on its own.
  • An empirical audit of ICML, NeurIPS, and ACL publications from 2024–2025 finds that synthetic data is widely used in privacy-sensitive settings while verification remains rare.The audit is presented as one of the paper’s central contributions.
  • The paper maps existing tools to the identified failure modes and distills them into a Minimum Privacy Claim Standard.This standard is presented as part of the paper’s contribution alongside the conceptual analysis and empirical audit.

2. The Conceptual Shift: From Formal Privacy to Empirical Privacy

Privacy has shifted from an adversarial-risk claim grounded in threat models and inference bounds toward an appearance-based property inferred from synthetic outputs. An audit of major ML venues finds synthetic data routinely used in privacy-sensitive contexts while explicit verification remains uncommon.

  • From Privacy as Risk to Privacy as Appearance: Earlier privacy claims specified an adversary and bounded individual-level inference rather than relying on released-data properties.Residual risk depends on auxiliary information and inference methods.
  • From Privacy as Risk to Privacy as Appearance: Synthetic text is often treated as inherently private because it is fluent, diverse, and rarely verbatim, despite lacking explicit threat models or bounded inference claims.Surface nonidentifiability can be mistaken for meaningful protection.
  • Industry, Policy, and Legal Narratives Reinforcing the Shift: Commercial tools commonly frame synthetic outputs as anonymous by construction, while inexpensive checks and certifications do not bound membership, attribute, or distributional inference.Table 1 summarizes representative commercial claims; SOC 2 and ISO 27001 assess organizational security controls rather than synthetic-data privacy.
  • Industry, Policy, and Legal Narratives Reinforcing the Shift: Industry and legal classifications reinforce output-based privacy interpretations even as data-protection authorities reject blanket anonymity and emphasize residual inference risk.The resulting evaluation regime treats generation or transformation as sufficient more often than adversarial analysis.
  • Synthetic Data as an Implicit Privacy Claim in ML Research: A conservative audit of ICML, NeurIPS, and ACL papers from 2024–2025 screened synthetic-data use, privacy language, and formal or empirical mechanisms.The pipeline was designed to undercount appearance-based claims.
  • Synthetic Data as an Implicit Privacy Claim in ML Research: Synthetic data appears consistently across venues and years, making it a routine component of contemporary ML workflows rather than an exceptional technique.The analysis distinguishes explicit construction from weaker indirect forms such as bootstrapped or pseudo-labeled supervision.
  • Synthetic Data as an Implicit Privacy Claim in ML Research: Across synthetic-using papers, privacy language frequently appears without explicit verification, while only a small minority pair privacy statements with formal guarantees or adversarial evaluation.At NeurIPS, near-universal privacy language is largely attributable to mandatory ethics and reproducibility statements.
  • Synthetic Data as an Implicit Privacy Claim in ML Research: The observed pattern is descriptive: privacy is increasingly implied by generation rather than specified through assumptions, threat models, or failure modes.The conference-guidance gap concerns how privacy claims should be supported when synthetic data is used in sensitive contexts.

3. Consequences of Not Correcting the Shift

Treating synthetic outputs as private by appearance produces structural false negatives, redistributes exposure toward rare records, and makes privacy claims difficult to compare, challenge, or enforce. These consequences affect both technical auditing and institutional accountability.

  • Systemic Consequences: Appearance-based privacy creates a systemic failure mode in which auditing, harm assessment, governance, and accountability organize around generation rather than adversarial risk.The resulting regime can appear functional while losing the ability to detect, measure, and respond to harm.
  • False Negatives Become the Norm: Synthetic data can pass reproduction and similarity audits while retaining training dependence detectable through reconstruction error, density structure, or local overfitting.These signals can persist without identifiable or verbatim copies.
  • False Negatives Become the Norm: Positive similarity and distance-based audit results do not bound membership inference because scalable attacks exploit distributional training dependence rather than verbatim reproduction.Such audits certify surface properties while statistically detectable training traces remain accessible.
  • Risk is Redistributed, Not Eliminated: Synthetic generation redistributes privacy risk toward rare or distinctive records because models favor high-density patterns and standard audits emphasize average-case behavior.Aggregate utility, similarity, and diversity metrics can mask subgroup-level exposure.
  • Risk is Redistributed, Not Eliminated: Low-frequency records and minority-associated attribute combinations show higher membership-inference exposure even when global utility and similarity metrics appear favorable.Higher generation fidelity can strengthen fine-grained signals that enable attribute or population-level inference.
  • Risk is Redistributed, Not Eliminated: Overall, synthetic data reduces exposure for typical cases while increasing exposure in the tails without making that redistribution explicit.The paper characterizes this as uneven and largely undetected privacy loss.
  • Institutional Consequence: Privacy Becomes Non-Actionable: When privacy is grounded in generation rather than explicit guarantees or threat models, claims become non-falsifiable and lack shared criteria for residual risk.Privacy becomes a descriptive label rather than an enforceable property.
  • Institutional Consequence: Privacy Becomes Non-Actionable: Narrow, self-selected audits can produce technically correct but substantively uninformative conclusions because their criteria exclude singling-out or membership inference.Standardized independent audits and mandatory reporting are proposed to link disclosure to explicit assumptions and evidence.

4. Re-grounding Privacy Claims in Synthetic Data

The paper re-grounds synthetic-data privacy in explicit, interpretable evidence about residual risk rather than in generation alone. It recommends using established tools to specify threats, scope protections, test inference, and report heterogeneous exposure.

  • Re-grounding Privacy Claims: Synthetic data may reduce direct data-access concerns but does not eliminate privacy risk or address inference without additional evaluation.The paper distinguishes usefulness of synthetic data from the validity of privacy claims made about it.
  • Explicit Risk Bounding: Differential privacy expresses protection as a parameterized bound on adversarial inference under stated assumptions.Its framing illustrates how privacy can be scoped as a claim about residual risk rather than a transformation property.
  • Empirical Inference Evaluation: Membership and attribute-inference attacks, including auditor-oriented frameworks such as TAPAS, directly test whether sensitive signals remain recoverable from synthetic outputs.Benchmarking can measure leakage continuously across generators and settings.
  • Risk Heterogeneity Analysis: Risk-heterogeneity analysis examines whether exposure concentrates on rare records or remains inferable through group-level attributes despite benign average-case metrics.Disaggregated reporting aligns evaluation with unevenly distributed privacy risk.
  • Re-grounding Privacy Claims: These tools do not replace synthetic data; they target specific failure modes and make privacy claims interpretable without requiring one protection strategy.Protocols that omit them remain scientifically valid but should not inherit implicit privacy guarantees from synthetic-data use.
  • Privacy Claims are Earned, not Implied: Authors should distinguish synthetic data used as a risk-reduction heuristic from synthetic data offered as evidence supporting a privacy claim.Only the latter requires the minimum claim conditions.
  • Privacy Claims are Earned, not Implied: Privacy claims supported by synthetic data should specify the threat model, define addressed and unaddressed risks, and provide at least one evidentiary mechanism.Examples include a stated-ε DP guarantee, an inference audit, or disaggregated risk reporting.
  • Privacy Claims are Earned, not Implied: The Minimum Privacy Claim Standard asks authors to state existing tools and assumptions explicitly without introducing new infrastructure.The proposal mirrors evidentiary norms already adopted elsewhere in ML research.

5. Alternative Views

The paper considers pragmatic risk reduction, negative evidence, and disclosure as alternative perspectives on synthetic-data privacy. It agrees that synthetic data can reduce exposure and that disclosure is useful, but maintains that neither operational utility, absent observed leakage, nor disclosure alone establishes privacy.

  • Synthetic data as pragmatic risk reduction: Synthetic data can reduce direct exposure and enable development or limited sharing under access constraints without providing formal privacy guarantees.The paper accepts this operational role but distinguishes it from using synthetic data as evidence for privacy claims.
  • Synthetic data as pragmatic risk reduction: The paper’s concern is not risk-mitigating use itself, but interpreting that heuristic as evidentiary when privacy is invoked as a scientific motivation.
  • Negative evidence as sufficient in practice: Negative empirical results can be informative and tractable, but treating absent observed leakage as resolving privacy concerns makes residual uncertainty implicit.
  • Disclosure norms instead of verification: Clear disclosure is necessary, but disclosure alone cannot specify which risks are reduced or distinguish their types and degrees.
  • The paper’s boundary is that privacy claims grounded in synthetic data require explicit risk accounting to preserve both synthetic data’s usefulness and privacy’s meaning.

6. Call to Action

The paper calls for lightweight ecosystem changes that distinguish synthetic data used for exposure reduction from synthetic data used as privacy evidence. It recommends explicit risk accounting and rejects categorical interpretations of synthetic data as privacy-safe.

  • Research-level norms: Authors should state whether synthetic data is a risk-reduction heuristic or evidence supporting a privacy claim, and whether privacy risk is inferred or evaluated.
  • Institutional and deployment norms: Organizations and regulators should resist treating synthetic data as categorically privacy-safe and require claims to specify unaccounted risks.
  • Scope and definitions: Privacy claims for synthetic data need clear articulation because interpretations vary across research, deployment, and governance contexts and protection needs differ across data types.
  • Alternative boundary: If explicit verification is rejected, synthetic data should be treated as access-enabling or exposure-reducing practice without invoking privacy-related scientific or institutional claims.

A. Verification and Privacy Guarantees of Synthetic Data Tools

The appendix supplies evidence for the privacy-verification categories in Table 1 by examining how commercial synthetic-data tools define privacy, whether their claims are verifiable, and what guarantees they offer.

  • The appendix summarizes commercial tools’ privacy definitions, claim verifiability, and levels of offered privacy guarantees.

A.1. Evidence from Vendor Websites

Vendor materials present varied privacy controls and claims, ranging from infrastructure assurances and privacy metrics to configurable differential privacy. The evidence also includes defaults, audit availability, and documented limitations.

  • Evidence scope: The vendor summaries rely on documentation and public webpages available at the time of writing, so offerings and guarantees may change.
  • Tonic.ai: Vendor documentation states that Tonic.ai’s differential privacy is disabled by default and that enabling it may remove rare categories.
  • Tonic.ai: Tonic.ai offers a configurable per-column DP toggle with ε = 1 by default in some generators, but no external audit is available.
  • Hazy: Hazy measures privacy risk with density-disclosure and presence-disclosure metrics and membership-inference simulations, but uses internal-only metrics without external audit.
  • YData: YData claims privacy compliance and re-identification protection through automated controls, inference attacks, divergence metrics, and configurable differential-privacy modes, without external audit or certification.
  • Gretel: Gretel advertises differential-privacy models, tunable privacy with mathematical guarantees, and a privacy dashboard, with no documented third-party audit.
  • Syntho: Syntho emphasizes downstream model quality and safety, including matching performance between synthetic- and real-data-trained models.
  • MOSTLY AI: MOSTLY AI emphasizes no one-to-one relation and default privacy settings while reporting SOC 2 and ISO 27001 certification, but no differential privacy.

Interpretation of Verification and Guarantee Categories

The verification categories distinguish unsupported privacy claims from internal checks, partner-supported evaluations, and third-party assessments. These categories vary in evidentiary strength and do not uniformly provide data-level privacy guarantees.

  • Some vendors make no mention of privacy tests or audits, leaving their privacy claims unsupported by reported verification.The category is explicitly illustrated as having no tests or audits mentioned.
  • Claims relying solely on the generative process provide no empirical verification of privacy loss.Tonic.ai is given as an example without differential privacy.
  • Unverified claims may invoke differential privacy or risk metrics without independent audits or supporting evidence.Hazy and SAS exemplify this category.
  • Internal tests can assess privacy and utility through divergence metrics or inference attacks but lack external validation.YData is listed as an example.
  • Internal-plus-partner approaches combine built-in privacy filters or scores with deployment on trusted platforms.Gretel on Google Cloud is the cited example.
  • Third-party evaluation covers data quality and/or privacy, while external evaluation is distinguished from formal auditing.Syntho evaluated by SAS illustrates external evaluation that is not a formal audit.
  • Organizational security certifications such as SOC 2 and ISO 27001 do not guarantee data-level privacy.MOSTLY AI is given as an example.
  • Tools such as DataCebo’s SDV can support empirical verification of privacy loss.

B. Construction of Conference-Level Measurements

The appendix defines a reproducible corpus of accepted ICML, NeurIPS, and ACL papers for descriptive conference-level measurements. It documents inclusion procedures without evaluating individual papers’ correctness, safety, or adequacy.

  • The appendix documents data sources, inclusion criteria, and counting procedures to make conference-level measurements auditable and reproducible.
  • The measurements characterize synthetic data’s positioning relative to privacy-related language and remain descriptive rather than evaluative.They do not assess the correctness, safety, or adequacy of individual papers.
  • The corpus contains accepted papers from ICML, NeurIPS, and ACL, with Table 2 reporting totals by venue and year.
  • All accepted papers were included across presentation formats for ICML and NeurIPS, while ACL included all accepted proceedings papers.
  • The corpus represents a defined fraction of each conference’s accepted papers based on structured acceptance labels and publicly accessible proceedings.
  • ACL papers for 2024 and 2025 were collected from official ACL Anthology proceedings because ACL does not use OpenReview acceptance labels.
  • No topic, research-area, methodology, contribution, or motivation filters were applied.

B.1.2. TOTAL PAPERS RETRIEVED

The corpus is compared with official proceedings counts, and synthetic-data usage is identified through a tiered keyword framework. The derived indicator includes explicit synthetic-data references and contextual implicit or programmatic indicators, while separating simulation-only papers.

  • The used corpus is largely aligned with official proceedings sizes, with minor differences potentially attributable to keyword-based paper linking.
  • Synthetic-data usage is detected through case-insensitive regular-expression matching over full paper texts using tiered keywords.
  • Synthetic-data detection: Tier S3 captures explicit references to fully synthetic or model-generated datasets, treating listed phrases as direct dataset-generation claims.Examples include “synthetic data,” “generated dataset,” and “LLM-generated data.”
  • Synthetic-data detection: Tier S2 captures weaker but substantive indicators such as pseudo-labeling or programmatic generation when they occur near a data noun.
  • Synthetic-data detection: Broad S1 mentions are recorded for completeness but excluded from aggregate synthetic-data usage statistics.Contextual requirements prevent generic generation references unrelated to dataset construction from being classified as usage.
  • Synthetic-data detection: A paper is classified as using synthetic data when either the S2 or S3 indicator is detected.
  • Simulation distinction: Simulation language is separately detected and marked as simulation-only when no S2 or S3 synthetic indicators are present.This prevents conflating simulated environments with synthetic datasets.

B.4. Privacy-Related Variables

The privacy-related variables distinguish general privacy claims from formal mechanisms, adversarial audits, scrubbing, and access controls. A higher-level label identifies appearance-based privacy claims grounded in synthetic generation without those safeguards.

  • Privacy claims include language about privacy preservation, anonymization, de-identification, regulation, personal data, sensitive attributes, or the absence of real user data.
  • Formal privacy captures mechanisms and accounting terms including differential privacy, DP-SGD, privacy budgets, epsilon, delta, PATE, and related accountants.
  • Privacy audits are identified through adversarial evaluations such as membership inference, model inversion, reconstruction, attribute inference, singling out, linkability, exposure, canaries, or extraction.
  • Scrubbing captures dataset-level sanitization, including PII removal, redaction, regex or NER-based de-identification, and related safety-filter combinations.
  • Access-control protections include secure enclaves, federated access, on-device processing, access controls, trusted execution environments, and data-use agreements.
  • Privacy by generation labels papers that use synthetic data and invoke privacy claims without formal mechanisms, privacy audits, or access controls.The label operationalizes appearance-based privacy claims grounded solely in data generation.
Loading 2609.01273v1…