Source-linked AI summary
ChatGPT is not all you need. A State of the Art Review of large Generative AI models
Roberto Gozalo-Brizuela, Eduardo C. Garrido-Merchan
TL;DR
Generative AI models can create novel content across formats, motivating a review of their growing industrial and societal relevance. The paper organizes recent models into a taxonomy, describes representative systems, and concludes that they show creativity, personalization, and accuracy while remaining limited by data, computation, bias, understanding, and ethics.
Problem
The rapid emergence of generative AI models and their effects across industries and society motivate a concise review and taxonomy of recent systems.
Method
The paper reviews representative generative AI models and organizes them into nine categories based on multimedia input-output mappings.
Results
The reviewed models demonstrate creativity and personalization in text-to-image and text-to-audio tasks, and accuracy in text-to-science and text-to-code tasks.
Takeaways & Limitations
Generative AI can help optimize creative and non-creative tasks across economies.
Takeaways & Limitations
Current models require difficult-to-obtain datasets, enormous parameters and computation, and remain challenged by out-of-distribution problems, bias, limited understanding, and ethical risks.
Abstract
from arXiv · showhide
During the last two years there has been a plethora of large generative models such as ChatGPT or Stable Diffusion that have been published. Concretely, these models are able to perform tasks such as being a general question and answering system or automatically creating artistic images that are revolutionizing several sectors. Consequently, the implications that these generative models have in the industry and society are enormous, as several job positions may be transformed. For example, Generative AI is capable of transforming effectively and creatively texts to images, like the DALLE-2 model; text to 3D images, like the Dreamfusion model; images to text, like the Flamingo model; texts to video, like the Phenaki model; texts to audio, like the AudioLM model; texts to other texts, like ChatGPT; texts to code, like the Codex model; texts to scientific texts, like the Galactica model or even create algorithms like AlphaTensor. This work consists on an attempt to describe in a concise way the main models are sectors that are affected by generative AI and to provide a taxonomy of the main generative models published recently.
1 Introduction
Generative AI creates novel content rather than merely analyzing existing data, using large datasets and powerful models to transform inputs into generated outputs. The review argues that these capabilities are reshaping industries while enhancing, rather than replacing, human work.
- Concept and scope: Generative AI generates novel content instead of simply analyzing or acting on existing data.Unlike predictive systems focused on classification or regression, generative models transform input information and generate new content.
- Model foundations: Modern generative models use large datasets and high-capacity architectures to represent complex, high-dimensional data distributions.The review identifies deep neural networks, transformers, generative adversarial networks, and variational autoencoders as examples of such architectures.
- Industry implications: The review is motivated by generative AI’s effects on industries including art and universities, where activities may need to adapt to generated text and artistic content.The authors frame the review as guidance for professionals working in industries that may benefit from these models.
- Industry implications: The authors argue that generative AI will enhance human content by inspiring artists and improving professors’ generated content rather than replacing humans.This claim is presented as the review’s view of how generative models will affect human work.
2 A Taxonomy of Generative AI models
The paper organizes recent generative AI models into a taxonomy based on multimedia input-output mappings and reviews models published mainly during 2022. It also observes that deployment is concentrated among a small number of organizations with substantial computational and technical resources.
- Taxonomy: The taxonomy contains 9 categories representing mappings between multimedia input and output data types.Each model shown in the taxonomy is described in the paper’s subsequent sections.
- Taxonomy: The covered models were selected to represent recent advances in generative AI and are organized by their input and generated formats.The taxonomy is introduced before the paper analyzes individual models in detail.
- Developers and resources: Only six organizations are behind deployment of the reviewed models.The paper attributes this concentration to the need for enormous computation power and highly skilled data science and data engineering teams.
- Release timing: All covered models were released during 2022 except LaMDA, released in 2021, and Muse, released in 2023.The release-date distribution is summarized in Figure 2.
- Developers and resources: The paper notes that model development includes acquired startups and collaborations between companies and academia.Examples include collaborations associated with Stable Diffusion, Soundify, and DreamFusion.
3 Generative AI models categories
The review covers nine generative AI categories and illustrates models spanning text-to-image and other input-output mappings. Its text-to-image examples use prompt-conditioned architectures including CLIP-based generation, transformer encoders, diffusion models, and latent diffusion.
- Categories: The paper analyzes nine generative AI categories introduced in its taxonomy.Each category is covered through the models presented in the taxonomy.
- Text-to-image models: DALL·E 2 generates original and realistic images or art from text descriptions using CLIP-based representations and a prior model.The prior generates possible CLIP image embeddings from a text caption, which are decoded into images.
- Text-to-image models: Imagen uses a pretrained text encoder and cascaded conditional diffusion models to map text embeddings into images at increasing resolutions.The review reports that scaling the language model improves sample fidelity and image-text alignment more than scaling the image diffusion model.
- Text-to-image models: Stable Diffusion is an open-source latent-diffusion model that supports image modification through operations in latent space.The model consists of a text encoder and an image generator.
Muse
Muse is a text-to-image transformer designed to generate high-quality images efficiently through masked modeling in discrete token space and parallel decoding.
- Muse: Muse uses masked modeling in discrete token space, requiring fewer sampling iterations than diffusion or autoregressive approaches.Its discrete-token design contributes to inference efficiency.
- Muse: Muse uses parallel decoding and is reported to be 10x faster than Imagen-3B or Parti-3B at inference time.The reported comparison concerns inference speed.
- Muse: Muse is reported to be 3x faster than Stable Diffusion v1.4 while both models operate in the latent space of a VQGAN.The comparison is specifically with Stable Diffusion v1.4.
- Dreamfusion: DreamFusion generates 3D models from text using a pretrained 2D text-to-image diffusion model and a differentiable generator.The differentiable generator helps produce 3D objects that render well from random angles.
- Muse: The review contrasts Muse with DALL·E 2 and Imagen through generated-image examples associated with corresponding text prompts.The figure places prompt text alongside model outputs for visual comparison.
- Magic3D: Magic3D addresses DreamFusion’s long processing time and low generated-image quality with a two-stage optimization framework.It builds a low-resolution diffusion prior, accelerates optimization with a sparse 3D hash grid, and further optimizes a textured 3D mesh.
3.3 Image-to-Text models
Image-to-text models convert visual inputs into textual responses or captions, using multimodal architectures that combine visual analysis with language generation.
- Flamingo: Flamingo accepts sequences of text tokens interleaved with images or videos and produces text responses to visual queries.Its architecture combines a vision model for scene analysis with a large language model for basic reasoning.
- Flamingo: Flamingo answers questions using the image included in the prompt, as illustrated by examples in Figure 10.Each example contains an input prompt with images and text followed by generated text.
- VisualGPT: VisualGPT is an OpenAI image-captioning model that leverages knowledge from the pretrained GPT-2 language model.It uses an encoder-decoder attention mechanism with an unsaturated rectified gating function to bridge modalities.
- VisualGPT: VisualGPT’s lower data requirement is presented as useful for rapid data curation, rare-object description, and specialized-domain applications.The passage identifies data efficiency as the model’s biggest advantage.
- VisualGPT: Figure 11 shows image-derived text prompts together with attention scores assigned to the words in those texts.For one example, “cat” and “television” receive the most discriminative information from the image.
Phenaki
Phenaki generates realistic videos from sequences of textual prompts and can also manipulate an input image according to text, while video generation remains computationally constrained.
- Phenaki: Phenaki generates realistic videos from sequences of textual prompts and supports open-domain time-variable prompts.It is described as the first model in the review capable of generating videos from open-domain time-variable prompts.
- Phenaki: Phenaki jointly trains on a large image-text dataset and a smaller set of video-text examples to generalize beyond available video data.The approach exploits the much larger scale of image-text datasets.
- Phenaki: Phenaki’s limitations include computational constraints when generating videos of variable length.The review also notes that video-text datasets are much smaller than image-text datasets.
- Phenaki: Phenaki uses a C-ViViT encoder, training transformer, and video generator to produce temporally coherent and diverse videos conditioned on text.The encoder compresses videos, while temporal and spatial transformers process the token representations before projection back to pixel space.
- Phenaki: Figure 12 presents image sequences generated by Phenaki from four different prompts.The figure illustrates text-conditioned sequence generation.
- Phenaki: Figure 13 shows Phenaki manipulating an input image according to a text prompt.The examples depict prompt-conditioned changes to the supplied image.
Whisper
The review describes Whisper as an audio-to-text transformer supporting multilingual speech recognition, translation, and language identification, trained on diverse internet audio.
- Whisper: Whisper performs multilingual speech recognition, translation, and language identification as an audio-to-text model.Its stated goal is reliable operation across a broad range of environments without supervised decoder fine-tuning for each deployment distribution.
- Whisper: Whisper is trained on 680,000 hours of labeled audio collected from the internet.The dataset spans diverse environments, recording setups, speakers, and languages, and is restricted to human voice.
- Whisper: The training files are divided into 30-second segments paired with the transcript subset occurring within each segment.This passage describes the segmentation and transcript pairing procedure.
- Whisper: Whisper uses an encoder-decoder transformer architecture that the review describes as validated to scale reliably.The paper’s figure is said to show the architecture’s data types and learning sequence.
- Whisper: The review places Whisper among text-output models that convert non-structured data into another representation to support different tasks.The surrounding discussion transitions from non-text modalities to models that output text.
LaMDA
LaMDA is a dialogue-specialized transformer language model that generates, filters, grounds, and reranks responses, while related systems support collaborative writing and brain-speech decoding.
- LaMDA: LaMDA is a dialogue-specialized transformer family with up to 137B parameters, pretrained on 1.56T words of dialogue data and web text.Fine-tuning can support safety and factual grounding, using only 0.001% of training data.
- LaMDA: LaMDA generates multiple responses, filters them for safety, grounds them in external knowledge, and reranks them for quality.
- Collaborative writing: The collaborative writing model decomposes writing into Plan, Edit, Explain, and Repeat steps, allowing human intervention throughout the process.The cycle continues until the text requires no further updates.
- Collaborative writing: Wikipedia edit histories support self-training, but noisy comments and missing citations require retrieval that does not always work.
- Brain-speech decoding: The brain-speech model uses contrastive learning and self-supervised speech representations to align noninvasive brain recordings with speech sounds.It uses EEG and MEG recordings from volunteers listening to audiobooks.
- Brain-speech decoding: Decoding performance improves with additional EEG and MEG recordings, but the research remains focused on speech perception rather than speech production.
3.7 Text-to-Code models
Text-to-code models address programming tasks, with Codex targeting general code translation and AlphaCode targeting complex unseen problems through larger data, architectures, and sampling.
- Scope: The review places code generation alongside scientific-text generation as a major application area for generative AI.
- Codex: Codex translates natural-language descriptions into code and is designed for general-purpose programming tasks.It particularly assists with mapping decomposed problems to existing libraries, APIs, and functions.
- AlphaCode: AlphaCode targets code-generation problems requiring deeper reasoning because other language models perform poorly on complex unseen problems.
- AlphaCode: AlphaCode combines an extensive training and evaluation dataset, efficient transformer architectures, and large-scale model sampling.
- AlphaCode: AlphaCode is pretrained on 715.1 GB of GitHub code and fine-tuned with Codeforces data for validation.
Galactica
The review situates scientific reasoning and algorithm discovery among additional generative-AI applications, highlighting Minerva’s step-by-step solutions and AlphaTensor’s reinforcement-learning search.
- Minerva: Minerva is a language model for mathematical and scientific questions that generates solutions through calculations and symbolic manipulation.It performs step-by-step reasoning without external tools such as a calculator.
- AlphaTensor: AlphaTensor discovered a more efficient matrix-multiplication algorithm, with implications for neural-network and scientific-computing workloads.
- AlphaTensor: AlphaTensor uses deep reinforcement learning to search tensor decompositions in a finite factor space through a single-player TensorGame.The score reflects the number of operations needed to obtain the correct multiplication result.
- Other models: GATO uses one network and shared weights across multimodal, multitask, and multi-embodiment capabilities including gameplay, image captioning, dialogue, and robotics.
- Other models: Other models extend generative AI to human-motion generation and slide creation using ChatGPT as a surrogate model.
4 Conclusions and further work
The review finds generative AI capable across creative and technical tasks, while emphasizing substantial data, computation, accuracy, bias, and ethical constraints.
- Capabilities: Generative AI demonstrates creativity, personalization, and accuracy across text-to-image, text-to-audio, text-to-science, and text-to-code tasks.
- Implications: The review suggests these capabilities can optimize both creative and non-creative economic tasks.
- Limitations: Training and operating these models requires enormous datasets, many days, advanced computers, and substantial computation capacity.
- Limitations: Models struggle more with solutions outside their training datasets, limiting performance on novel problems.
- Limitations: Data bias remains a major issue requiring control, despite Galactica’s attempt to address it with a no-bias layer.
- Limitations: Text-to-science accuracy remains below the level professionals would need for routine day-to-day reliance.
- Limitations: Generative models require ethical constraints because text-to-video systems can be used to create deep-fakes.