PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs Meet Multimodal Generation and Editing: A Survey

Yingqing He, Zhaoyang Liu, Jingye Chen, Zeyue Tian, Hongyu Liu, Xiaowei Chi, Runtao Liu, Ruibin Yuan, Yazhou Xing, Wenhai Wang, Jifeng Dai, Yong Zhang, Wei Xue, Qifeng Liu, Yike Guo, Qifeng Chen

arXiv:2405.19334v2cs.AIcs.CLcs.CVcs.MMcs.SD

TL;DR

Existing multimodal world-simulator models may generate video without generating or perceiving other modalities. This survey reviews LLM-augmented generation and editing across modalities and concludes that LLMs contribute to the quality and capabilities of generative systems.

  • Problem

    Sora can generate realistic videos but cannot generate or perceive other modalities, limiting the multimodal scope described for a world simulator.

    Source: §1 · p. 1

  • Method

    The survey systematically reviews LLM-augmented generation and editing across images, video, 3D, and audio, including the roles LLMs play.

    Source: §1 · p. 2

  • Results

    The review highlights significant contributions of LLMs to the quality and capabilities of generative systems.

    Source: §12 · p. 36

  • Takeaways & Limitations

    The authors anticipate further innovation at the intersection of AI and generative content toward a more unified and capable multimodal generation framework.

    Source: §12 · p. 36

    Instruction-training methods face high LLM training costs and a trade-off between performance and training costs, while generated instruction-corpus quality is difficult to control.

    Source: §8.2.2 · p. 28

Abstract

from arXiv · show

With the recent advancement in large language models (LLMs), there is a growing interest in combining LLMs with multimodal learning. Previous surveys of multimodal large language models (MLLMs) mainly focus on multimodal understanding. This survey elaborates on multimodal generation and editing across various domains, comprising image, video, 3D, and audio. Specifically, we summarize the notable advancements with milestone works in these fields and categorize these studies into LLM-based and CLIP/T5-based methods. Then, we summarize the various roles of LLMs in multimodal generation and exhaustively investigate the critical technical components behind these methods and the multimodal datasets utilized in these studies. Additionally, we dig into tool-augmented multimodal agents that can leverage existing generative models for human-computer interaction. Lastly, we discuss the advancements in the generative AI safety field, investigate emerging applications, and discuss future prospects. Our work provides a systematic and insightful overview of multimodal generation and processing, which is expected to advance the development of Artificial Intelligence for Generative Content (AIGC) and world models. A curated list of all related papers can be found at https://github.com/YingqingHe/Awesome-LLMs-meet-Multimodal-Generation

1 INTRODUCTION

The survey examines how LLMs can support language-guided generation and editing across modalities, motivated by the goal of multimodal world simulators that can perceive and respond to the physical world. It reviews prior and emerging approaches, the roles LLMs play, and the field’s scope, safety topics, applications, and future directions.

1 INTRODUCTION

  • The survey targets a world-simulator gap: Sora can generate realistic video but cannot generate or perceive other modalities such as text, 3D, and audio.Source: §1 · p. 1
  • It presents a systematic review of LLM-augmented generation and editing for images, videos, 3D, and audio.Source: §1 · p. 2
  • The survey compares generative techniques before and after LLMs and summarizes LLM roles in generation or editing across modalities.Source: §1 · p. 2
  • It also discusses generative AI safety, emerging applications, and future directions for multimodal generation and world models.Source: §1 · p. 2

1.1 Scope

  • The scope covers separate and joint generation of images, video, 3D, and audio, primarily for language-based generation and editing; pure text and unconditional generation are not primary focuses.Source: §1.1 · pp. 2–3

1.2 Content Overview

  • The survey reviews generative and alignment techniques, LLMs and MLLMs, modality-specific generation, multimodal agents, generative AI safety, applications, and future directions.Source: §1.2 · p. 3

2 RELATED SURVEYS

Earlier surveys cover generation by individual modality or review multimodal understanding and generation together. This survey focuses on how LLMs contribute to generation across modalities.

Survey on modality-specific generation

  • Modality-specific surveys cover image, video, 3D, or audio generation, while this survey examines LLMs’ roles across modalities.Source: §2 · p. 3

Survey on LLMs and MLLMs

  • Prior LLM and MLLM surveys mix understanding with generation and focus less on generation, whereas this work centers on generation’s performance and functionality.Source: §2 · p. 3

3 PRELIMINARIES

This section introduces generative models, multimodal alignment, LLMs, and MLLMs, outlining their core principles and components.

3.1 Generative Models

  • Generative models map prior latents to data-like samples; GANs use a generator and discriminator in a two-player minimax game.The discriminator distinguishes real from generated samples while the generator aims to fool it.Source: §3.1 · pp. 3–4, §3.1.1 · p. 4

3.1.2 Variational Auto-Encoder

  • VAEs encode inputs into latent distributions and decode them back, optimizing an ELBO; VQ-VAE adds a learned discrete codebook for more structured latents.The VAE encoder typically models the latent distribution as Gaussian.Source: §3.1.2 · p. 4

3.1.3 Flow-based Model

  • Flow-based models use sequential invertible transformations to map data toward a simpler prior, enabling exact computation of the negative log-likelihood objective.The transformations are chosen to be easily invertible with easily computable Jacobians.Source: §3.1.3 · p. 4

3.1.4 Diffusion Model

  • Diffusion adds scheduled Gaussian noise, then learns reverse denoising by predicting it; the Markov noise assumption simplifies training, and mean-squared error trains predictions.Denoising proceeds backward through the time steps.Source: §3.1.4 · pp. 4–5

3.1.5 Autoregressive Model

  • Autoregressive models predict future values from past values and train by maximizing the likelihood of the next value given the preceding sequence.Their negative log-likelihood objective can be computed exactly because of the model’s sequential structure.Source: §3.1.5 · p. 5

3.2 Multimodal Alignment Model

  • Multimodal alignment methods learn shared representations across modalities: CLIP links image and text, CLAP links text and audio, CAVP links video and audio, and ImageBind aligns six modalities.ImageBind’s modalities include text, image, video, audio, depth, and thermal.Source: §3.2 · p. 5

3.3 Large Language Models

  • LLMs use transformer decoders to predict the next word from left context, with large-scale pretraining followed by task-specific fine-tuning.The section also notes CoT fine-tuning and RLHF as techniques to improve task-specific abilities.Source: §3.3 · p. 5

3.4 Multimodal Large Language Models

  • MLLMs add modality-specific encoders and projectors to align multimodal features with an LLM; generation-capable systems add output projectors and modality generators.Some systems train alignment modules around a pretrained LLM, while others train the full multimodal system end to end.Source: §3.4 · p. 5

4 IMAGE GENERATION AND EDITING

The section traces image generation from domain-limited synthesis through text-guided open-domain models to interactive LLM-based generation, then surveys image editing methods and their datasets.

4.1 Image Generation

  • Image generation has progressed from narrow categories to prompt-driven open-domain synthesis and, more recently, conversational iterative generation.Early systems targeted categories such as faces or bedrooms; DALL-E [158] and LDM [30] enabled open-domain prompt-based generation.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.1.1 Text-guided Image Generation with CLIP

  • CLIP aligns text and images for prompt-guided generation, while later work explores how LLMs can further benefit image generation.LLM-based work includes unified visual-token understanding and generation, as well as methods that enhance pretrained text-to-image models.Source: §4.1.1 · p. 6

4.1.2 Text-guided Image Generation with LLMs

  • MLLM approaches generate interleaved text and images, support multimodal instruction following, or connect LLMs to pretrained image generators.Examples include DreamLLM’s multi-round dialogue and interleaved generation, and adapters that connect LLMs with Stable Diffusion XL.Source: §4.1.2 · pp. 6, 8, 9

4.1.3 Image Layout Planning via LLMs

  • LLM layout planners translate prompts into structured object positions, counts, and sizes to address text rendering and spatial-relation challenges in generated images.Methods include CSS-style layouts, scene layouts, and iterative box corrections; TextDiffuser-2 plans text position and content.Source: §4.1.3 · p. 9

4.1.4 Prompt Synthesis and Refinement via LLMs

  • LLMs can expand or refine prompts, select suitable generators, and evaluate generated images for text alignment or aesthetic quality.DreamSync selects images using separate text-alignment and aesthetic assessments, while LLMScore checks image descriptions against instructions.Source: §4.1.4 · p. 9, §4.1.5 · p. 9

4.2.1 Image Editing with CLIP/T5

  • CLIP/T5-based editing includes prompt-embedding tuning and tuning-free diffusion approaches, but CLIP’s capabilities limit many methods to simple instructions.SDEdit adds noise to an input image and denoises it under text guidance; other tuning-free methods manipulate latents, cross-attention maps, or UNet features.Source: §4.2.1 · p. 10

4.2.2 Image Editing with LLMs

  • LLM-based editing spans instruction-data construction, dialogue-driven facial edits, expressive-instruction guidance, and object-level correction of generated images.CHATEDIT tracks edit intentions across dialogue; MGIE is evaluated on Photoshop-style manipulations, global photo optimization, and local editing.Source: §4.2.2 · pp. 10–11

4.3 Image-language datasets

  • Image-generation datasets range from captioned everyday-scene collections to web-scale image-text pairs and instruction-following multimodal data.LAION-5B contains 5.85 billion image-text pairs; LLaVA Visual Instruct 150K was generated via a GPT model.Source: §4.3 · p. 11

5 VIDEO GENERATION AND EDITING

The section surveys video generation and editing across CLIP/T5-based and LLM-based approaches, covering generation paradigms, LLM roles, editing methods, and video-language datasets.

5.1 Video Generation

  • Video generation uses diffusion or autoregressive transformer models; pixel-level diffusion offers better text alignment but requires more computation than efficient latent-level approaches.The survey groups video methods by technical components and datasets, and distinguishes CLIP/T5-based from LLM-based approaches.Source: §5.1.1 · p. 11, §5.1 · p. 11, Table 5 · p. 12

5.1.2 Text to Video Generation with LLMs

  • LLM-based video generation uses multimodal autoregressive models, scene layouts, and temporally structured prompts to guide video synthesis.VideoPoet targets temporal consistency and high-motion fidelity; MAGVIT-v2 tokenizes visual inputs and reports benchmark performance, compression, and representation-learning results.Source: §5.1.2 · p. 12

5.1.3 Video Layout Planning via LLMs

  • LLM-generated layouts, bounding boxes, scene graphs, or simulated-scene representations guide spatial placement and motion consistency in video generation.GPT4Motion converts Blender scenes into intermediate conditions such as depth maps for Stable Diffusion.Source: §5.1.3 · p. 12

5.1.4 Temporal Prompt Generation via LLMs

  • LLMs expand prompts into frame descriptions, coherent prompt sequences, contact plans, posture scripts, or multi-scene scripts for temporal guidance.DirecT2V decomposes a single prompt into frame-by-frame descriptions, while Free-Bloom sequences prompts for evolving visual content.Source: §5.1.4 · pp. 12–13

5.2.1 Text-guided Video Editing with CLIP/T5

  • CLIP-based video editing includes one-shot tuning, attention-map manipulation, and key-frame editing propagated across frames to address temporal consistency.Tune-A-Video has poor temporal stability and can alter unrelated regions; Pix2Video edits a key frame first and propagates the edit.Source: §5.2.1 · p. 13

5.2.2 Text-guided Video Editing with LLMs

  • LLM-based video editing remains limited and largely follows the approach of constructing synthetic training data for instruction-guided editing.InstructVid2Vid and InsV2V use synthetic instruction data with Stable Diffusion and conditional 3D U-Nets to produce temporally coherent edits.Source: §5.2.2 · p. 13

5.3 Video-language Datasets

  • Video-language datasets vary from manually annotated descriptions and event-localized captions to large-scale web videos with weak or model-generated captions.MSR-VTT contains 200k clip-language pairs from 10K videos; Vript has 12K annotated videos with fine-grained shot, camera, content, and scene information.Source: §5.3 · pp. 13–14

6 3D GENERATION AND EDITING

The section contrasts LLMs and CLIP/T5 models as ways to connect text with 3D assets for generation, editing, and understanding.

6 3D GENERATION AND EDITING

  • LLMs support iterative updates to 3D generation and understanding according to user requirements, facilitating human-computer interaction.The section gives language-guided human-motion generation as an example.Source: §6 · p. 14
  • Unlike LLMs’ direct iterative influence on assets, CLIP/T5 integrates rendered-image and text features to inject textual information into 3D assets.Source: §6 · p. 14
  • The survey covers CLIP- or LLM-guided 3D generation, editing, and understanding, with a generic pipeline and an overview table of methods.Source: §6 · p. 14

6.1 3D Generation

The survey reviews CLIP/T5- and LLM-based approaches to 3D generation, from text-guided optimization and dataset supervision to natural-language interaction with 3D assets. It also describes limitations of CLIP-guided generation and the distinct roles LLMs play in generating and understanding 3D content.

6.1.1 3D Generation with CLIP/T5

  • The survey identifies three typical ways to provide text information during 3D generation: CLIP supervision, distillation from pretrained text-to-image models with SDS, and annotated 3D datasets for training.Source: §6.1.1 · pp. 14–15
  • CLIP-guided methods generate and edit meshes, NeRFs, shapes, motion, and multimodal shape descriptions from text or other inputs.Examples include text-guided mesh stylization, NeRF control, shape completion, and motion-language modeling.Source: §6.1.1 · p. 15
  • CLIP’s semantic-level loss can reduce high-frequency spatial information, and text-guided 3D methods continue to face visual artifacts.The survey identifies this as an active research challenge for improving visual quality and fidelity.Source: §6.1.1 · p. 15
  • SDS-based methods distill 3D assets from pretrained text-to-image models, with later work targeting quality and optimization limitations.Reported limitations include over-saturation, over-smoothing, multi-face Janus artifacts, and time-consuming computation; proposed approaches include VSD, acceleration, and Gaussian splatting.Source: §6.1.1 · pp. 16–17
  • Fantasia3D separates geometry and texture generation, refining a normal map before predicting a PBR color field with geometry fixed.The components can be manipulated independently by artists and designers.Source: §6.1.1 · p. 16
  • Dataset-supervised approaches train on annotated 3D assets or multiview images, with some adapting 2D text-to-image diffusion models for 3D prediction.Point-E [313] and Shape-E [352] train on point clouds, while MVDream [345], RichDreamer [348], SPAD [346], and UniDreamer use Objaverse multiview images and text and camera-pose guidance.Source: §6.1.1 · p. 17

6.1.2 3D Generation with LLMs

  • LLMs have been introduced to enhance human-computer interaction in 3D generation and support generating, manipulating, or understanding 3D assets through textual instructions.Source: §6.1.1 · p. 17, §6.1.2 · p. 17
  • LLM-based 3D methods translate natural-language requests into Blender scripts or support interaction with 3D assets through instruction answering, point-cloud analysis, and varied 3D tasks.3D-GPT [82] uses three agents to produce Blender code, while SceneCraft [353] iteratively improves scripts and builds a reusable skill library.Source: §6.1.2 · p. 17

6.2 3D Editing

The survey organizes 3D editing into CLIP/T5-based methods and LLM-based approaches. CLIP/T5 methods support targeted scene, geometry, and texture edits, while the survey reports no dedicated LLM-based editing method.

6.2 3D Editing

  • The section frames 3D editing as two approaches, first discussing CLIP/T5-based methods.It then distinguishes LLM-based editing as a separate subsection.Source: §6.2 · p. 17, §6.2.2 · p. 18

6.2.1 3D Editing with CLIP/T5

  • CLIP/T5-based editing uses text or sketches to modify existing NeRF scenes, meshes, and local semantic regions.Methods include CLIP-guided region editing and stylization, mesh deformation, sketch-guided generation, and text-instructed NeRF or mesh editing.Source: §6.2.1 · pp. 17–18

6.2.2 3D Editing with LLMs

  • The survey reports no specific LLM-based 3D editing method; editing is treated as a subtask of LLM-based generation.It notes that 3D-GPT [82] and SceneCraft [353] can edit assets directly and says future developments will be followed.Source: §6.2.2 · p. 18

6.3 Summary

The survey presents CLIP and LLM integration with 3D assets as a route to more intuitive human-computer interaction. Textual information supports natural-language control and helps users interpret and fine-tune generated or edited 3D outputs.

6.3 Summary

  • Text input lets users express 3D generation and editing intentions without relying as heavily on specialized software or technical expertise.The survey also says textual information can improve output interpretability and explainability, helping users understand and fine-tune results.Source: §6.3 · p. 18
  • The survey identifies computer graphics, virtual reality, and augmented reality as potential application areas for text-aligned 3D generation and editing.It describes these advances as having potential to enhance user experiences and support creative work.Source: §6.3 · p. 18

7.1 Domains

The survey organizes LLM-related audio work across general sounds, music, and speech, covering generation, understanding, and editing. It describes LLMs as backbones, conditioners, labellers, agents, or inspiration for audio systems.

7.1 Domains

  • The audio survey covers general sounds, music, and speech across generation, understanding, and editing, and summarizes technical approaches and datasets.It points to Table 9 for technical components, Table 8 for datasets, and Fig. 8 for milestone works.Source: §7.1 · p. 18, Figure 8 · p. 20, Table 8 · p. 20

7.1.1 General Audio Sounds

  • General audio understanding models use LLMs as backbones to process speech, environmental events, and music, with Qwen-Audio spanning over 30 tasks.The survey also describes automated audio captioning and LLM-coordinated tool interfaces for audio understanding.Source: §7.1.1 · pp. 18, 20
  • General audio systems combine text embedding or parsing with latent diffusion, discrete audio-token modeling, flow matching, or unified next-token prediction.Examples include TANGO, Make-an-Audio 2, AudioLM, Audiobox, and UniAudio; WavJourney coordinates external audio models through an LLM agent.Source: §7.1.1 · pp. 20–21

7.1.2 Music

  • Music understanding approaches include feature analysis, refined instruction annotations, tool-using agents, and multilingual zero-shot lyric transcription.The survey contrasts MU-LLaMA’s music-feature analysis, LLARK’s refined annotations, and MusicAgent’s tool-based interaction; LyricWhiz combines Whisper and GPT-4 for transcription.Source: §7.1.2 · p. 22
  • Music generation methods range from text-conditioned diffusion with controls for melody, dynamics, and rhythm to symbolic music generation using ABC notation or MIDI.M2UGen handles diverse music, image, and video inputs; ChatMusician and SongComposer use different symbolic representations and formats.Source: §7.1.2 · p. 23

7.1.3 Speech

  • Speech understanding work uses LLM-based backbones for multimodal speech processing, ASR, and inter-modal knowledge transfer.SpeechGPT [86] introduces the SpeechInstruct dataset; AudioPaLM [89] combines PaLM-2 [402] and AudioLM for speech understanding and generation.Source: §7.1.3 · p. 23
  • Speech generation includes codec-token language modeling for text-to-speech and unified audio-text models for recognition, understanding, and generation.VALL-E [55] reframes TTS as conditional language modeling, while LauraGPT [91] handles audio and text tasks.Source: §7.1.3 · pp. 23–24

7.2 Roles of LLMs

The survey groups audio-generation roles for LLMs as backbone, conditioner, labeller, agent, and LLM-inspired backbone. Across these roles, LLMs can coordinate audio processing and generation, condition outputs, expand dataset descriptions, or model audio tokens.

7.2 Roles of LLMs

  • Audio LLM roles are grouped as backbone, conditioner, labeller, agent, and LLM-inspired backbone.The survey frames language as an abstraction that helps audio research approach generative understanding.Source: §7.2 · p. 24

7.2.1 LLMs as Backbone

  • Cascade backbones pair an LLM with modality-specific encoders or decoders, while unified approaches tokenize audio and expand the LLM vocabulary to support a scalable uniform structure.LTU and Pengi use audio representations as input to predict paired text; M2UGen adds an audio decoder for generation. The survey notes cascade training may reduce efficiency and scalability when data and compute are abundant.Source: §7.2.1 · p. 24

7.2.2 LLMs as Conditioner

  • As a conditioner, an LLM encodes text prompts to guide audio generation; MusicGen tests found T5 achieved the highest subjective text-relevance score among the tested encoders.Tango uses Flan-T5, latent diffusion, a Mel-Spectrogram VAE, and a vocoder to produce audio from a text prompt.Source: §7.2.2 · p. 24

7.2.3 LLMs as Labeller

  • LLMs can turn class labels in audio datasets into full-sentence descriptions, with manually templated descriptions paraphrased through self-instruction to enrich the dataset.The survey identifies audio captioning as the transformation from labels to sentence-form audio descriptions.Source: §7.2.3 · pp. 24–25

7.2.4 LLMs as Agent

  • As agents, LLMs coordinate audio tools through text-based task analysis, model assignment, and response generation, as illustrated by AudioGPT and MusicAgent.AudioGPT converts outputs from audio foundation models into text for LLM interaction; MusicAgent combines tools for tasks such as music classification, separation, and lyric recognition.Source: §7.2.4 · p. 25

7.2.5 LLMs Inspired Backbone

  • LLM-inspired audio backbones model discretized audio tokens; adding semantic tokens from self-supervised representations improves semantic consistency in unconditional speech continuation.VALL-E uses residual vector quantized acoustic tokens to continue speech from short audio and text prompts while preserving speaker characteristics; AudioLM adds semantic tokens after acoustic-token-only training lacked semantic-level consistency.Source: §7.2.5 · p. 25

8 TOOL-AUGMENTED MULTIMODAL AGENTS

The survey reviews multimodal agents that extend LLMs with external tools for interactive generation and editing. It describes planning, execution, and response stages, contrasts training-free and instruction-tuned planning, and illustrates capabilities and constraints.

8 TOOL-AUGMENTED MULTIMODAL AGENTS

  • Tool-augmented LLMs are presented as a growing human-computer interaction direction, enabling interactive generation or editing of images, videos, and audio through external tools.The review focuses on works extending LLMs to multimodal generation by augmenting them with tools.Source: §8 · p. 25

8.1 Motivation

  • Multimodal agents address LLMs' inability to generate or edit other modalities by connecting natural-language requests to tools across image, video, and audio.The section motivates this approach as supporting more natural and versatile interaction and creative applications.Source: §8.1 · pp. 25–26

8.2 Methods

  • The general framework plans tool invocations and their arguments, executes external multimodal tools, then generates a user-facing response from execution outputs.Existing methods differ chiefly in task planning and are divided into training-free and instruction-tuning approaches.Source: §8.2 · p. 26

8.2.1 Training-free Methods

  • Training-free agents use prompts and in-context learning; examples include modular program execution, ReAct-style alternating thought and tool use, and HuggingGPT's separate task parsing and model selection.The survey notes these methods can fail on complex problems, require manual prompt design, depend on LLM tool-use capability, and may lose performance as the tool set grows.Source: §8.2.1 · p. 27

8.2.2 Instruction-tuning Methods

  • Instruction-tuned agents train LLMs for tool use, with approaches spanning self-instructed visual-tool data, tool retrieval with memory control, task decomposition over tool graphs, and multimodal API prediction.GPT4Tools constructs 71.4K instruction-response pairs covering 31 visual tools; MLLM-Tool uses ImageBind-based embeddings for six modalities. The survey flags training cost, corpus quality, and generalization to unseen instructions as limitations.Source: §8.2.2 · p. 28

8.3 Demonstrations

  • Demonstrations show image generation, editing, and understanding; in one comparison, ControlLLM's image aligned better with the instruction, while HuggingGPT returned two images after decomposing the request.Pointing input can select an object with SAM for targeted editing; the survey also reports video generation and dubbing, audio generation, and weather querying, but no agent with direct video-frame editing.Source: §8.3 · p. 29

8.4 Summary

  • The survey concludes that multimodal agents support more natural interaction and creative applications, while recognizing that challenges remain.It summarizes the approaches as training-free prompting or instruction tuning within a planning, execution, and response-generation framework.Source: §8.4 · p. 29

9 GENERATIVE AI SAFETY

The survey reviews generative-AI safety work addressing harmful or biased outputs, copyright, and fabricated content. It covers attacks, defenses, attribution and watermarking, and datasets for evaluating unsafe content.

9 GENERATIVE AI SAFETY

  • Safety research targets biased and toxic generation, copyright protection, and harms from fabricated generative content.The survey describes vulnerabilities to malicious use as a distinct challenge for generative models.Source: §9 · p. 29
  • Reported attack approaches include optimization-based adversarial methods, prompt manipulations that bypass safeguards, and data poisoning that exploits compromised training-data integrity.The survey cites attacks affecting model performance, bias or harmful outputs, and audio and video systems.Source: §9 · p. 29
  • Defenses include detection or prompt-context controls without changing model parameters, alongside preference-based alignment methods such as PPO and DPO.Latent Guard checks text-to-image prompts in latent space; the survey describes DPO-related approaches as more efficient than PPO for learning alignment from preference data.Source: §9 · pp. 29–30
  • Deepfake detection methods look for visual artifacts, but the survey says their reliance on patterns seen in training limits performance on new AIGC models.The cited methods focus on distinguishing deepfake videos.Source: §9 · p. 30
  • Data attribution and watermarking are discussed as copyright measures, while safety datasets evaluate unsafe content across text and image settings.Examples include SafetyBench, with 11,435 items across seven categories, and GOAT-Bench, with more than 6K topics for unsafe memes.Source: §9 · p. 30

10 APPLICATIONS

The survey reviews applications of LLMs and multimodal generation across image, video, audio, and 3D, as well as cross-domain uses. Its examples range from prompt-based creation and editing tools to digital humans and generative tools used with people in professional fields.

10.1 Image

  • Image applications include prompt-based image generation and editing, open-source model fine-tuning, and LLM-assisted prompt refinement.The survey highlights DALLE3's integration with ChatGPT4 for text-prompted image generation and modification.Source: §10.1 · p. 31

10.2 Video

  • Video tools let users generate clips from text, with applications including short-film assembly, human video, and efforts to reduce film and television production costs.The survey lists commercial and open-source examples and notes progress in realism and prompt following after Sora demo videos were released.Source: §10.2 · p. 31

10.3 Audio

  • Audio applications span speech, sound transfer, music, transcription, and editing, with potential uses in education, dubbing, intelligent terminals, voice assistance, and medicine.Examples include Descript's transcription-based editing and text-prompted song generation tools.Source: §10.3 · p. 31
  • 3D applications include digital-human creation, reconstruction from 2D images, and natural-language-based scene reconstruction, while text-to-3D remains an active research topic.The survey also notes 3D uses in film, games, design, architecture, and virtual reality.Source: §10.4 · pp. 31–32

10.4.1 Others

  • Cross-domain applications include AI-generated movies combining 3D, video, music, and speech, digital humans, and multimodal generative tools in mathematics, law, education, and robotics.The survey characterizes these as emerging applications of multimodal generative models with LLMs.Source: §10.4.1 · p. 32

11 FUTURE PROSPECTS

The survey sees LLM-augmented multimodal generation as a route to better quality, diversity, controllability, and interactivity. Its future-prospects discussion identifies technical priorities spanning fidelity, sequence length, control, consistency, unified training, efficiency, and safety.

11 FUTURE PROSPECTS

  • The authors frame LLM-augmented multimodal generation as promising for improving content quality, diversity, controllability, and interactivity.The technical prospects aim to inform future work.Source: §11 · p. 32, §11.1 · p. 32

11.1.1 High-resolution Generation

  • High-resolution generation matters for detailed, realistic content, but its substantial hardware and time costs make efficient generation an additional research priority.The discussion spans image, video, audio, and 3D generation.Source: §11.1.1 · p. 32

11.1.2 Long-term Sequence Generation

  • Long video and audio sequences must maintain continuity, avoid repetition, and introduce novel elements aligned with the input and overarching theme.The survey suggests multimodal fine-tuning and prompt engineering or conditioning as possible approaches.Source: §11.1.2 · pp. 32–33

11.1.3 More Accurate and Fine-grained Generation Control

  • Current control methods can miss fine details such as fingers or misread control signals, while stronger language models may improve text rendering and control alignment.The survey points to T5-XXL as an encoder associated with better spelling in generated images.Source: §11.1.3 · p. 33

11.1.4 Multi-view Consistency

  • Multi-view consistency remains difficult amid occlusions, lighting changes, and geometric distortions; methods also struggle to generalize and render complex or textureless geometry.Examples include Sculpt3D’s sparse 3D prior and HarmonyView’s consistency–diversity sampling trade-off.Source: §11.1.4 · p. 33

11.1.5 Unified Training for Multimodal Generation

  • Unified multimodal training must address cross-modal feature alignment, interference among modality-specific objectives, and computational overhead.The survey also notes early efforts toward generating multimodal content within one model.Source: §11.1.5 · p. 33

11.1.6 Efficient Training and Deployment Strategies

  • Efficient training and deployment remain open needs as models and datasets scale; low-rank approximation and mixed precision can impair quality or stability.Quantization can reduce inference memory and computation, but may lower accuracy and output fidelity.Source: §11.1.6 · p. 34

11.1.7 Ethically Safe Content Generation

  • More capable video generation heightens societal concerns around deepfakes, while security research remains fragmented across detection, alignment, and post-hoc checking.The survey proposes considering defenses against attacks that transfer from open-source to commercial closed-source models.Source: §11.1.7 · p. 34

11.2 Application Prospects

The survey sketches applications for multimodal generative models, from semantic audio and storytelling to interactive design, 3D scenes, and customizable avatars. It also connects these capabilities to prospective world-model uses in education, filmmaking, and virtual worlds.

11.2.1 Semantic Audio Synthesis

  • Semantic audio synthesis can use text, images, or video as contextual cues for generating corresponding audio and personalized auditory experiences.Proposed application areas include entertainment, virtual reality, and assistive technologies.Source: §11.2.1 · pp. 34–35

11.2.2 Multi-modal Storytelling.

  • Multimodal storytelling can start from text or images and generate complementary narratives, image sequences, video, or audio.The survey describes both prompt-to-sequence and image-to-video-or-audio directions.Source: §11.2.2 · p. 35

11.2.3 Interactive Content Design

  • Interactive content design lets creators rapidly explore and refine media in real time, potentially reducing manual work and lowering barriers to creative tools.The survey also points to future opportunities in VR, AR, and immersive storytelling.Source: §11.2.3 · p. 35

11.2.4 3D Scene Generation

  • Text descriptions, sketches, blueprints, or design parameters could guide generation of interactive 3D environments and architectural renderings.The survey presents these as ways to support rapid iteration and exploration of design options.Source: §11.2.4 · p. 35

11.2.5 Customizable Avatars

  • Multimodal models could create personalized avatars with generated appearance and voice, while further work could support expressive behavior and virtual-environment integration.The discussion includes customization of facial features, clothing, hairstyle, expression, and accessories.Source: §11.2.5 · p. 36

11.3 Towards the World Model

  • The survey links world models to perceptual modalities including vision, audition, and speech, as well as spatial understanding and generation.It highlights possible applications in multimodal education and communication, movie generation, and the metaverse.Source: §11.3 · p. 36

12 CONCLUSION

The survey concludes by reviewing LLM-augmented multimodal editing and generation across image, video, 3D, and audio, alongside multimodal agents and safety concerns. Its examples illustrate agents producing multiple outputs from an instruction and other multimodal generation and editing interactions.

12 CONCLUSION

  • The survey reviews LLM-augmented multimodal editing and generation across image, video, 3D, and audio, and discusses generative agents and safety.The authors hope the review will inform future multimodal generation, particularly world models.Source: §12 · p. 36
  • The accompanying examples include image generation, click-based image editing, image-conditioned audio generation, and weather visualization.The examples also show dialogue prompts for image and audio generation.Source: Figure 12 · p. 37, §12 · pp. 37–38, Figure 13 · p. 38
  • One agent example turns a text instruction into an image, then uses that generated image to create a video.The instruction requests an image of an insect robot preparing a meal followed by a video based on the image.Source: Figure 15 · p. 39, §12 · p. 39
Loading 2405.19334v2…