Source-linked AI summary
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT
Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip S. Yu, Lichao Sun
TL;DR
AIGC has advanced rapidly across generative tasks and applications, creating a need to synthesize its core technologies, applications, and open problems. This survey reviews generative-model history, unimodal and multimodal AIGC, common techniques, applications, trustworthiness concerns, and future directions.
Problem
AIGC’s rapid progress and adoption make an extensive review of past research and open problems necessary.
Method
The survey synthesizes generative-model history, foundation components, unimodal and multimodal models, applications, techniques, trustworthiness, and future directions.
Results
The survey provides a comprehensive overview of AIGC’s history, recent advancements, applications, techniques, trustworthiness concerns, open problems, and future directions.
Takeaways & Limitations
The survey aims to provide readers with a comprehensive understanding of recent generative-AI developments and future challenges.
Takeaways & Limitations
AIGC raises unresolved social concerns involving bias, ethics, and effects on stakeholders, including potential discrimination and inequities.
Abstract
from arXiv · showhide
Recently, ChatGPT, along with DALL-E-2 and Codex,has been gaining significant attention from society. As a result, many individuals have become interested in related resources and are seeking to uncover the background and secrets behind its impressive performance. In fact, ChatGPT and other Generative AI (GAI) techniques belong to the category of Artificial Intelligence Generated Content (AIGC), which involves the creation of digital content, such as images, music, and natural language, through AI models. The goal of AIGC is to make the content creation process more efficient and accessible, allowing for the production of high-quality content at a faster pace. AIGC is achieved by extracting and understanding intent information from instructions provided by human, and generating the content according to its knowledge and the intent information. In recent years, large-scale models have become increasingly important in AIGC as they provide better intent extraction and thus, improved generation results. With the growth of data and the size of the models, the distribution that the model can learn becomes more comprehensive and closer to reality, leading to more realistic and high-quality content generation. This survey provides a comprehensive review on the history of generative models, and basic components, recent advances in AIGC from unimodal interaction and multimodal interaction. From the perspective of unimodality, we introduce the generation tasks and relative models of text and image. From the perspective of multimodality, we introduce the cross-application between the modalities mentioned above. Finally, we discuss the existing open problems and future challenges in AIGC.
1 INTRODUCTION
AIGC uses generative AI to create content from human instructions, with recent progress driven by larger models, datasets, and computation. This survey reviews AIGC history, techniques, applications, and open challenges across unimodal and multimodal generation.
- Definition and Motivation: AIGC products such as ChatGPT and DALL-E-2 have attracted broad societal attention, with DALL-E-2 generating images from textual descriptions.The paper presents image generation from instructions as an example of AIGC.
- Definition and Motivation: AIGC generates digital content from human instructions by extracting intent information and producing content that satisfies the instruction.The process consists of intent extraction followed by content generation.
- Definition and Motivation: Recent advances reflect larger datasets, foundation models, and computational resources, improving models’ ability to extract human intent.GPT-3 increased pre-training data from 38GB to 570GB after filtering and model size from 1.5B to 175B compared with GPT-2.
- Survey Scope and Contributions: The survey covers foundation techniques, recent unimodal and multimodal generation advances, applications, risks, open problems, and future directions.Its contributions include a formal definition, historical review, analysis of GAI tasks and models, and discussion of challenges and trends.
- Survey Organization: The survey is organized around generative-model history, basic training components, recent unimodal and multimodal advances, applications, risks, and future directions.Multimodal topics include vision-language, text-audio, text-graph, and text-code models.
2 HISTORY OF GENERATIVE AI
Generative AI developed through distinct advances in language and vision before transformer architectures connected modalities. The survey traces this history from early statistical models and hand-designed methods to modern prompting and multimodal representation learning.
- Early Generative Models: Generative models date to the 1950s, when Hidden Markov Models and Gaussian Mixture Models generated sequential data such as speech and time series.Deep learning later brought significant performance improvements to generative modeling.
- Natural Language Generation: Early NLP generation used N-gram language models, while recurrent neural networks later enabled modeling of relatively long dependencies.N-gram methods learned word distributions and searched for the best sequence, but struggled with long sentences.
- Computer Vision Generation: Before deep learning, computer-vision generation relied on hand-designed texture synthesis and texture mapping, limiting complex and diverse image generation.GANs emerged in 2014 as a major milestone, followed by VAEs and diffusion models.
- Convergence Across Modalities: Transformer architecture became a shared backbone across NLP and computer vision, supporting multimodal models such as CLIP.Transformers were introduced for NLP in 2017 and later applied to vision, where visual components enabled image-based applications.
- Recent Techniques: Recent techniques include few-shot prompting in NLP and self-supervised contrastive learning for more robust visual-language representations.Few-shot prompting includes selected dataset examples in the prompt to help models understand task requirements.
3 FOUNDATIONS FOR AIGC
AIGC foundations combine transformer-based architectures, large-scale training infrastructure, and alignment methods such as RLHF. These components support scalable generation while addressing human-intent alignment and practical training constraints.
- Foundation models: Transformer architectures underpin many state-of-the-art AIGC models, using self-attention to route information across sequences and support parallelizable computation.The architecture contains encoder and decoder components, with multi-head attention and feed-forward layers.
- Pre-trained language models: Pre-trained language models are commonly divided into masked encoders, autoregressive decoders, and encoder-decoder models according to their training objectives.BERT represents masked language modeling, while GPT and OPT represent autoregressive modeling.
- Reinforcement learning from human feedback: RLHF aligns model outputs with human preferences through pre-training, reward learning from pairwise human comparisons, and reinforcement-learning fine-tuning.The reward model encodes human preferences and guides policy optimization.
- Open challenges: RLHF adoption remains constrained by limited public benchmarks and implementation resources for evaluating and fine-tuning language-model reinforcement learning.RL4LMs is presented as an open-source library intended to provide reusable building blocks.
- Training infrastructure: Large-scale AIGC training has been enabled by faster hardware, distributed training, and cloud access to clusters of GPUs or TPUs.Distributed systems split workloads across processors or machines, while cloud services provide computing resources on demand.
4 GENERATIVE AI
Generative AI models support unimodal text and image generation as well as multimodal transformations across modalities. The survey covers language-model families, vision-generation methods, and text-audio and text-graph generation.
- Unimodal generation: Unimodal generative models take text or images as input and generate predictions in the same modality.The survey identifies GPT-3, BART, and T5 as language models and GANs, VAEs, and normalizing flows as vision models.
- Generative language models: Pre-trained language models use encoder, decoder, or encoder-decoder structures for tasks including dialogue, translation, question answering, and generation.Decoder models are widely used for text generation, while encoder models are mainly applied to classification.
- Generative language models: InstructGPT improves GPT-3 through human demonstrations, ranked responses, reward-model training, and preference-based optimization.The procedure uses labeler demonstrations for supervised fine-tuning before reward-guided optimization.
- Vision generative models: GANs generate data with a generator that learns the real-data distribution and a discriminator that distinguishes generated samples from real data.Research variants modify structure, conditioning information, latent representations, and objective functions to improve control or stability.
- Vision generative models: VAEs learn probabilistic representations and reconstruct inputs, while normalizing flows transform simple distributions into complex ones through invertible differentiable mappings.VAE research includes complex priors, skip connections, latent-space regularization, and hierarchical organization.
- Multimodal generation: Text-graph generation converts between natural language and structured graph representations, including semantic forms, knowledge graphs, and fluent text.The surveyed tasks distinguish text-to-graph semantic parsing, text-to-KG generation, and KG-to-text generation.
5 APPLICATIONS
AIGC applications span conversational systems, art, music, programming, education, and broader research-area ecosystems. These applications use generative models to create or assist with varied digital content and tasks.
- Conversational applications: Chatbots use language models to understand user inputs and respond conversationally through text-based interfaces.Examples include customer support, frequently asked questions, and empathy-oriented systems such as XiaoIce.
- Application landscape: The survey organizes generative-AI applications across research areas, applications, and related companies.The corresponding relation graph distinguishes research areas, applications, and companies by circle color.
- Art: AI art generation uses trained algorithms to create original artwork, with diffusion-based products including DALL-E-2, DreamStudio, and Imagen.DreamStudio is described as generating images from phrases or sentences using stable diffusion.
- Music: Deep music generation creates novel pieces from learned representations such as piano rolls encoding timing, pitch, velocity, and instruments.AIVA is identified as a notable example developed by Aiva Technologies in 2016.
- Programming: AI-based programming systems address code completion, code transformation, program repair, API prediction, feedback, and natural-language-to-code generation.CodeGPT is presented as a transformer-based model fine-tuned for code tasks, while CodeParrot provides interactive programming assistance.
- Education: AIGC can support personalized education by combining multimodal resources such as tutorial videos and academic papers.Minerva is described as using PaLM with a science-and-mathematics-focused dataset for college-level quantitative tasks.
6 EFFICIENCY IN AIGC
AIGC efficiency concerns arise from increasingly large and costly foundation models, motivating prompt-based methods and model compression. Prompt learning reformulates prediction through templates, demonstrations, and answer mapping to improve task adaptation and robustness.
- Prompt learning: Prompt learning reformulates prediction by finding a template x′ that directly predicts P(y|x′) rather than using the original input x.Its process includes prompt engineering and answer engineering.
- Prompt learning: Prompt engineering uses either manually designed discrete prompts or continuous prompts added to input embeddings to convey task-specific information.The approach adapts the input representation to the target task before inference.
- Prompt learning: Answer engineering maps the language model’s generated answer back into the ground-truth label space after task reformulation.This mapping connects generated language to classification outcomes.
- Prompt learning: Multi-prompt methods ensemble prompts or provide answered prompt augmentations during inference to improve prediction robustness.These approaches are described as more effective than relying on a single prompt.
- Efficiency challenges: Large foundation models achieve strong results but create efficiency problems because of their size and computational expense.Inference efficiency concerns model size, speed, and resource consumption, while training efficiency concerns time, memory, scalability, and data requirements.
- Efficiency challenges: Model compression reduces model size, computation, memory, and resource requirements through parameter or structure compression.Parameter compression includes pruning, quantization, low-rank decomposition, and parameter sharing.
7 TRUSTWORTHY & RESPONSIBLE AIGC
Trustworthy and responsible AIGC remains constrained by factuality, toxicity, privacy, and security risks. The survey describes evaluation, safety-training, and attack-related research addressing these concerns.
- Security: AIGC can generate plausible but factually unreliable content, including counterfactual or absurd answers that threaten online information truthfulness.In one cited test, ChatGPT generated false narratives for 80 of 100 previously identified false arguments.
- Security: AIGC safety concerns include biased outputs, stereotypes, misinformation, and other forms of toxic content.The survey describes metrics based on illustrative human values and adversarial studies of model safety.
- Security: Models trained with reinforcement learning from human feedback scale better and become increasingly difficult to red team across tested model sizes and training schemes.The comparison included models with 2.7B, 13B, and 52B parameters.
- Privacy: Membership inference attacks seek to determine whether an image belongs to training data, with practical methods assessing posterior-estimation matching or loss and likelihood.The survey distinguishes realistic same-distribution settings from an easier different-distribution assumption.
- Privacy: Data extraction attacks pose stronger privacy risks than membership inference because they seek to retrieve training images resembling specific data examples.The survey links their feasibility to memorization in large-scale models.
8 OPEN PROBLEMS AND FUTURE DIRECTIONS
The survey identifies open problems spanning high-stakes reliability, specialization versus generalization, continual updating, reasoning, scaling, and social impacts. These challenges concern accuracy, transparency, resource use, adaptation, and responsible deployment.
- High-stakes Applications: High-stakes AIGC applications remain challenging because healthcare, finance, autonomous vehicles, and scientific discovery require high accuracy, reliability, transparency, and near-zero fault tolerance.The survey recommends confidence scores, reasoning, and source information alongside generated results.
- Specialization and Generalization: Foundation-model dataset choices require balancing specialization against generalization under testing-time distribution shift.Highly specialized dataset curation can be time-consuming and cost-ineffective.
- Continual Learning and Retraining: Continual learning does not always outperform retraining, leaving unresolved when each strategy should be selected for updating model knowledge.The survey also identifies modularized foundation-model design as a way to clarify which components require retraining.
- Reasoning: Large generative models can still fail at common-sense reasoning, while constructing task-specific chain-of-thought prompts remains unresolved.Chain-of-thought prompting has been applied to question answering, vision-language question answering, and code generation.
- Scaling up: Scaling up is limited by compute budgets, available datasets, and model size, while larger pretraining models substantially increase training time and resource requirements.Poorly designed model-size and data-volume hyperparameters can waste resources and prevent desired outcomes.
- Social issues: AIGC proliferation raises social concerns involving bias, ethics, and impacts on stakeholders, including possible amplification of existing societal biases.These concerns are especially prominent across natural-language and computer-vision applications.
9 CONCLUSION
The survey synthesizes AIGC history, techniques, applications, trustworthiness concerns, and future directions across unimodal and multimodal generative models. It aims to clarify recent developments and challenges in generative AI.
- Conclusion: The survey reviews AIGC history and recent advances across unimodal and multimodal generative models, applications, techniques, trustworthiness, and responsibility.It also discusses open problems and future directions.
- Conclusion: The survey’s objective is to provide readers with a comprehensive understanding of recent generative-AI developments and future challenges.Its analysis of the general AI-generation framework aims to distinguish contemporary generative models from predecessors.
A CURATED ADVANCES IN GENERATIVE AI
This section reviews recent significant advancements in generative AI by organizing models into unimodal and multimodal frameworks. It further categorizes models by modality and summarizes related paper details in tables.
- The survey reviews recent significant advancements in generative AI.
- Generative models are classified into unimodal and multimodal frameworks.
- Models are further categorized by modality, with related paper details summarized in tables.
A.1 Language
This section summarizes major milestone models in natural language processing, including their general architectures and backbones.
- The section summarizes major milestone models in natural language processing.
- The surveyed language-model architectures generally include probabilistic objectives, encoders, decoders, and encoder-decoder structures.
- The section also summarizes the backbones of these natural language methods.
A.2 Vision
This section provides a table of major vision generative models.
- Table 3 summarizes major vision generative models.
- The table concerns generative models for vision.
- The section presents the vision-model information in tabular form.
A.3 Vision Language
This section provides a table of major vision-language models.
- Table 4 summarizes major vision-language models.
- The table concerns models combining vision and language.
- The section presents the vision-language model information in tabular form.
A.4 Text Audio
The section presents tables of major models associated with text audio, graph, and code categories.
- Table 5 is labeled as covering major text audio models.
- Table 6 is labeled as covering major text graph models.
- Table 7 is labeled as covering major text code models.