Security and Privacy on Generative Data in AIGC: A Survey

1.1 Background

• AIGC generates high-quality images, videos, text, audio, and graphics rapidly, increasingly producing content that humans struggle to distinguish from real data.Its applications include visual generation, language interaction, and emerging digital and physical environments.Source: §1.1 · pp. 1–2

1.2 Motivation

• The proliferation of generative data creates privacy risks through training-data replication, authenticity risks through false or forged content, and regulatory demands for non-toxic and factual outputs.These concerns motivate protection of generative data across multiple stages of the AIGC process.Source: §1.2 · pp. 2–3

1.3 Comparisons with Existing Surveys

• Earlier surveys addressed selected AIGC threats, but differed in coverage of generative-data security, authenticity, access control, and compliance.The surveyed works respectively emphasized watermarking, responsible AIGC concerns, or technologies such as privacy computing and blockchain.Source: §1.3 · p. 3
• This survey targets generative data, analyzes security and privacy through privacy, controllability, authenticity, and compliance, and adds coverage of access control, generative detection, and generative data for privacy protection.It also identifies future directions toward trustworthy generative data.Source: §1.3 · p. 4

2.1.1 Real Data for Training.

• AIGC begins with diverse real-data collection, followed by filtering, preprocessing, augmentation, and task-dependent privacy protection to improve data quality and security.Sources include open-source repositories, public databases, social media, and online forums.Source: §2.1.1 · pp. 4–5

2.1.2 Generative Model in Training.

• Generative models learn patterns from training data and require substantial computational resources, while fine-tuning adapts pretrained models to new tasks or domains using additional data.Common architectures include GANs, VAEs, flow-based models, and diffusion models.Source: §2.1.2 · p. 5

2.1.3 Generative Data.

• After training, users provide conditions such as questions or text, and the model generates outputs including realistic images, videos, code, and articles.AIGC supports rapid text generation, conversational interaction, and multimodal content creation.Source: §2.1.3 · p. 5

2.1.4 Analysis for Generative Data.

• Generated data requires quality assessment for accuracy, consistency, and integrity, followed by model adjustment when deficiencies are identified.Risk analysis also checks for discriminatory, false, or misleading content so detected hazards can be addressed.Source: §2.1.4 · p. 5

2.2 Security and Privacy on Generative Data

• Privacy protects sensitive individual information, while generative data can both replicate sensitive training content and replace sensitive real data with virtual content.Thus, AIGC can create privacy threats or support privacy protection depending on how generated data is used.Source: §2.2.3 · p. 6
• Controllability requires managing information and access to prevent unauthorized actions, including misuse, bias, and copyright infringement.Access control limits unauthorized utilization, while traceability tracks generation and dissemination for accountability.Source: §2.2.4 · p. 6

2.2.3 Authenticity.

• Authenticity preserves data integrity, truthfulness, accuracy, and credible origin, requiring methods that distinguish AI-generated data from real data and attribute outputs to generative models.Generative detection informs people whether content was AI-generated, while attribution supports credibility and accountability.Source: §2.2.4 · p. 6

2.2.4 Compliance.

• Compliance requires adherence to laws, regulations, and industry standards, including non-toxicity and factuality in generative data.These requirements exclude toxic content and inaccurate or illogical information.Source: §2.2.4 · pp. 6–7

3 Privacy on Generative Data

• AIGC can replicate sensitive training data, while virtual content can replace real identities or sensitive regions to reduce privacy exposure.The survey distinguishes privacy in AIGC from AIGC for privacy: the former risks leakage through memorization, while the latter uses generated content to preserve utility with altered identities.Source: §3 · p. 7, §3.1.1 · p. 7

3.1.1 Threats to Privacy.

• Larger training datasets increase privacy risk, and language models can memorize and reproduce names, emails, and phone numbers through querying.Reported studies find that larger models memorize faster and that memorization scales log-linearly with model size, replication count, and token count.Source: §3.1.1 · p. 7
• Diffusion models show stronger memorization than GANs, reproducing individual images or recombining semantically identical foreground and background objects from training data.GAN replication declines with dataset size and complexity, whereas diffusion-model studies identify image memorization and retrieval-like recombination.Source: §3.1.1 · p. 8, Figure 4 · p. 8

3.1.2 Countermeasures to Privacy.

• Privacy countermeasures include memorization rejection, dataset deduplication, differential privacy, replication detection, and machine unlearning.These approaches respectively reject near-duplicates, remove repeated training content, add training noise, detect copied outputs, or fine-tune models to forget sensitive data.Source: §3.1.2 · pp. 8–9

3.2 AIGC for Privacy

• AIGC for privacy generates surrogate faces, adversarial faces, mobility data, recommendation interactions, medical images, aerial backgrounds, and IIoT data.These applications aim to protect sensitive content while preserving attributes, recognition utility, segmentation masks, analytical regions of interest, or accuracy.Source: §3.2 · p. 9, §3.2.1 · p. 9, §3.2.2 · p. 9

3.3 Summary

• Differential privacy provides a provable guarantee but can reduce distribution fidelity, while detection and deduplication avoid model manipulation but depend on suitable similarity metrics.The survey identifies machine unlearning as promising but still relatively underdeveloped for increasingly large generative models.Source: §3.3 · p. 10, Table 2 · p. 10
• Adversarial attacks can either prevent generative models from learning private content or undermine privacy-preserving methods such as machine unlearning.The latter attack path can prevent sensitive information from being removed and thereby worsen privacy risks.Source: §3.3 · p. 10
• Realism, diversity, and controllability support privacy protection for real data, especially unstructured data, but generated virtual data must avoid reproducing private training content.The survey notes that stronger diffusion-model generation and improving controllability may expand privacy-protection applications.Source: §3.3 · p. 10

4 Controllability on Generative Data

• Controllability is needed because uncontrolled generative data can enable copyright infringement and malicious use, while passive detection and attribution have limited effectiveness.The section therefore emphasizes proactive access control and post hoc traceability.Source: §4 · pp. 10–11

4.1 Access Control

• Adversarial perturbations can disrupt model access to real data while preserving data quality and allowing rapid restoration when removed.The approach controls model outputs or learning behavior at the source rather than relying only on after-the-fact mitigation.Source: §4.1 · p. 11
• Face-protection methods use nullifying perturbations, identity-changing cloaks, or two-tier perturbations that disrupt manipulation and can encode source-tracking information.The cited methods respectively nullify translation, map faces to new latent identities, or produce blurry outputs while supporting passive tracking.Source: §4.1 · p. 11, Figure 5 · p. 11
• Copyright-protection methods add minor perturbations or style cloaks to disrupt DreamBooth and diffusion-model training on artistic works.Other approaches optimize disruption across visual, textual, and shared representation spaces or estimate perturbations theoretically for diffusion models.Source: §4.1 · pp. 11–12

4.2.1 Watermarking.

• Watermarking supports copyright protection, authenticity detection, and accountability by embedding identifiable information in models, inputs, or generated data.Model-specific watermarking embeds information into generators, whereas data-specific watermarking modifies inputs whose marks persist into generated outputs.Source: §4.2.1 · pp. 12–13
• Diffusion models can embed control information through progressive denoising, and latent-decoder watermarking can produce invisible, robust binary signatures for detection and identification.Flexible watermark messages can be changed without retraining, while attempts to evade the message matrix degrade generated quality.Source: §4.2.1 · p. 12
• Trigger-based and universal watermarking schemes activate marks under specified prompts or inject a shared detector-compatible watermark into arbitrary pretrained generators.These designs support prompt copyright verification and reduce the need to retrain detectors for new generators.Source: §4.2.1 · p. 12

4.2.2 Blockchain.

• Blockchain-based frameworks record generation lifecycles, ownership, copyright, distribution, and incentives to support transparent traceability and accountable exchange.Proof-of-AIGC deregisters plagiarized content, while AIGC-Chain records the lifecycle of AIGC products.Source: §4.2.2 · pp. 13–14, Table 3 · p. 14

5.1 Threats to Authenticity

• Human observers misclassify generative images at a 38.7% error rate, while deepfakes enable identity swaps, fabricated political speech, and authentication fraud.Source: §5.1 · p. 15
• Existing detectors struggle with evolving generation patterns and large models; analyzed ChatGPT detectors reached up to 90% TNR while maintaining low TPR.Source: §5.1 · p. 15

5.2.1 Generative Detection.

• Visual detectors exploit generation artifacts, including GAN edge and texture defects and diffusion-model noise or detail distortions, using residual and content streams.Source: §5.2.1 · pp. 15–16
• Detection methods also use diffusion reconstruction error, intrinsic dimensionality, inter-pixel correlations, invariant real-image features, and sampling strategies to improve generalization.Source: §5.2.1 · pp. 16–17
• Model-based text detectors can significantly outperform metric-based detectors, while attribution extensions recognize source models but still have room for improvement.Source: §5.2.2 · p. 17

5.2.2 Generative Attribution.

• Attribution methods identify GAN or diffusion sources from unique fingerprints, with a hierarchical approach recognizing architectures at average accuracy above 97%.Source: §5.2.2 · pp. 17–18
• Training-data attribution identifies subsets of source images contributing most to generated data, supporting copyright protection for training data rather than models.Source: §5.2.2 · p. 18

5.3 Summary

• Fingerprints evolve as models are optimized, watermarking requires preprocessing, and adversarial attacks can either prevent inauthentic generation or obfuscate detector decisions.Source: §5.3 · p. 18

6.1 Requirements to Compaliance

• Compliance requires adherence to laws, regulations, and ethical standards spanning ethics, bias, and politics, with policies increasingly addressing fairness, privacy, accountability, disclosure, toxicity, and factuality.Source: §6.1 · p. 19

6.1.1 Non-toxicity.

• Unintervened training data can transfer sexuality, hatred, political, violent, and racial toxicity into generated content, while models exhibit gender, religious, and cultural biases.Source: §6.1.1 · p. 19
• Diffusion models can produce many toxic images under harmful prompts, and toxic-image generation can persist even when prompts are innocuous.Source: §6.1.1 · pp. 19–20

6.1.2 Factuality.

• Unfactual outputs can sound authoritative and include plausible explanations, creating risks for healthcare, journalism, and education when users rely on incorrect information.Source: §6.1.2 · p. 20

6.2.1 Countermeasures to Non-toxicity.

• Toxicity countermeasures include dataset filtering, generation guidance, model fine-tuning, and generated-result filtering, but filtering can reduce downstream performance and ablation definitions remain limited.Source: §6.2.1 · pp. 20–21

6.2.2 Countermeasures to Factuality.

• Factuality methods assess generated claims with relation classifiers, fact extraction, multidimensional metrics, hidden activations, debate, tool feedback, mistake analysis, and refusal training.Source: §6.2.2 · pp. 21–22
• Larger language models in the 126M–530B range showed higher factual accuracy, attributed to greater capacity for learning generalized knowledge.Source: §6.2.2 · p. 21

6.3 Summary

• Compliance standards should be defined by application scenario rather than blanket denial, while adversarial attacks can undermine both compliant generation and compliance detectors.Source: §6.3 · p. 22

7.1.1 Benchmark Dataset.

• HC3 provides tens of thousands of expert-labeled ChatGPT responses across domains, while MixSet combines AIGC and human text for real-world operations.These datasets address broad-domain evaluation and the gap between isolated generated-text corpora and mixed-content scenarios.Source: §7.1.1 · p. 22
• DiffusionDB contains 14 million Stable Diffusion images, providing the largest-scale visual generative dataset described for studying generative image protection.Its scale and diversity support research on visual generative data protection.Source: §7.1.1 · p. 23

7.1.2 Benchmark Evaluation.

• Benchmark evaluations cover privacy leakage, watermarking, deepfake detection, gender bias, and factuality through representative resources including PrivLM-Bench, WaterBench, DeepfakeBench, GenderCARE, and FELM.The benchmarks standardize comparisons and support reproducible evaluation across the four information-security properties.Source: §7.1.2 · p. 23

7.2 Statistical Analysis.

• Research from 2020 to July 2024 focuses mainly on traceability and generative detection, motivated by malicious uses of generative data and policy attention to controllability.The survey also identifies forensic and watermarking techniques as transferable to generative-data issues.Source: §7.2 · p. 23, Figure 10 · p. 24
• Generative data can serve as virtual data that replaces sensitive data in telemedicine, recommendation algorithms, and smart cities to avoid personal-privacy leakage.This positions privacy-preserving synthetic data as an application area alongside detection and traceability.Source: §7.2 · p. 23
• Non-toxicity and factuality account for a small share of studies, with compliance research focusing mainly on text rather than visual generative data.The survey expects compliance-related work to grow under policy guidance, although governance spans sociology, management, law, and computer science.Source: §7.2 · pp. 23–24

8.1 Privacy

• Machine unlearning and concept forgetting do not prove private-content removal, while differential privacy requires retraining and may sacrifice unacceptable usability.The survey proposes information-theoretic methods but notes that measuring retained information in generative models remains challenging.Source: §8.1 · p. 24

8.1.1 Provable Private Content Removal Based on Information Theory.

• Fine-grained privacy and utility metrics are difficult because generative-model memory mechanisms and multimodal visual information are complex.Semantic-level metrics may better balance privacy and utility, but generic metrics must accommodate scenario-dependent semantics.Source: §8.1.1 · p. 24

8.1.3 Exploratory Solutions to the Privacy Onion Effect.

• The privacy onion effect means removing the most vulnerable outlier layer can make a previously secure layer the greatest privacy risk.Outlier memorization may explain the effect, while clustering outliers could reduce dataset diversity and threaten utility.Source: §8.1.3 · p. 25

8.2 Controllability

• Existing access-blocking methods suppress all semantics, whereas practical control should restrict specified meanings or user roles while preserving permitted content.Achieving such fine-grained access control requires understanding the generative process and its intrinsic mechanisms.Source: §8.2.1 · p. 25

8.2.1 Fine-grained Access Control of Generative Data.

• Text watermarking must balance high capacity against robustness because text is sparse and semantically fragile, unlike visual data.Multi-bit watermarking can increase capacity but significantly reduces robustness.Source: §8.2.2 · p. 25

8.3 Authenticity

• Generative detectors generalize poorly to unseen models, so future methods should exploit stable real-data invariants such as signal, noise, and texture distributions.Interpretable invariant analysis could make detection more credible for judicial forensics.Source: §8.3.1 · p. 25

8.3.1 Generalizable Generative Detection based on Invariance of Real Data.

• Tamper detection remains difficult because generative data can be both AI-generated and AI-tampered, making model fingerprints unreliable.Semi-fragile watermarks are suggested but cannot be assumed because uniform post-hoc adoption by all users is unrealistic.Source: §8.3.2 · pp. 25–26

8.3.2 Tamper Detection of Generative Data.

• Detection tools should become interpretable, trustworthy, and easy to handle because machine-vision outputs currently provide only initial judgments for important decisions.User research and explainable AI are identified as routes toward tools usable by unskilled users and older adults.Source: §8.3.3 · p. 26

8.4 Compliance

• Assessing generative-data compliance is difficult because laws provide broad principles, data are multimodal and cross-modal, and requirements differ across countries and regions.Automated assessment therefore requires legal concretization, multimodal fusion, and sensitivity to geographical and cultural contexts.Source: §8.4.1 · p. 26

8.4.1 Quantification and Assessment of Compliance.

• Toxic content can evade human perception and conventional detectors yet remain recoverable by attackers using specialized extractors.Human-intelligence-guided toxicity detection is proposed to identify stealthy toxicity with less manual labor.Source: §8.4.2 · p. 26

• The survey reviews security and privacy issues in generative data and corresponding solutions from the perspective of fundamental information-security properties.Source: §9 · p. 26
• Its countermeasure analysis is organized around privacy, controllability, authenticity, and compliance, revealing successful experiences from state-of-the-art protection measures.Source: §9 · p. 27
• It concludes by identifying possible future directions for security and privacy in generative data and promoting trustworthy AIGC applications.Source: §9 · p. 27
• The survey guides AIGC designers to revise models for these properties and urges users to protect individual privacy and prohibit potentially non-compliant generation.Source: §9 · p. 27