Source-linked AI summary
Any-to-Any Generation via Composable Diffusion
Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, Mohit Bansal
TL;DR
Existing multimodal generators do not flexibly handle arbitrary input and output combinations, especially when aligned training data is unavailable. CoDi aligns modalities in conditioning and diffusion spaces to enable composable simultaneous generation, and experiments report strong multimodal quality alongside competitive unimodal synthesis. Its realistic synchronized outputs also raise deepfake and misinformation concerns.
Problem
Existing generative systems are limited to restricted modality combinations, while serial generation can be slow and produce inconsistently aligned multimodal streams.
Method
CoDi aligns modality representations in input conditioning and diffusion spaces, using cross-attention and a shared environment space for composable generation.
Results
CoDi demonstrates arbitrary multimodal generation, including unseen joint combinations, with strong quality across multimodal settings and competitive or state-of-the-art unimodal results.
Takeaways & Limitations
CoDi provides a flexible foundation for generating coherent single or multiple modalities from diverse input combinations.
Takeaways & Limitations
CoDi’s ability to generate realistic and synchronized multimodal outputs can be exploited to create convincing deepfakes and misinformation.
Abstract
from arXiv · showhide
We present Composable Diffusion (CoDi), a novel generative model capable of generating any combination of output modalities, such as language, image, video, or audio, from any combination of input modalities. Unlike existing generative AI systems, CoDi can generate multiple modalities in parallel and its input is not limited to a subset of modalities like text or image. Despite the absence of training datasets for many combinations of modalities, we propose to align modalities in both the input and output space. This allows CoDi to freely condition on any input combination and generate any group of modalities, even if they are not present in the training data. CoDi employs a novel composable generation strategy which involves building a shared multimodal space by bridging alignment in the diffusion process, enabling the synchronized generation of intertwined modalities, such as temporally aligned video and audio. Highly customizable and flexible, CoDi achieves strong joint-modality generation quality, and outperforms or is on par with the unimodal state-of-the-art for single-modality synthesis. The project page with demonstrations and code is at https://codi-gen.github.io
1 Introduction
CoDi addresses the limitations of serial, modality-specific generation by processing arbitrary input combinations and simultaneously generating arbitrary output combinations. It aligns modalities in conditioning and diffusion spaces to avoid exhaustive combination-specific training while preserving coherent multimodal generation.
- Motivation: Existing cross-modal generators are limited when multiple modalities must interact, because serial pipelines are slow and independently generated streams may be misaligned.This is especially problematic for synchronized outputs such as video and audio.
- Contribution: CoDi is designed to simultaneously process and generate arbitrary combinations of modalities, including text, image, video, and audio.
- Method: Aligning modalities in both input conditioning and the generation diffusion step addresses scarce or nonexistent training data for many modality combinations.
- Method: Bridging Alignment enables efficient modeling of exponentially many input-output combinations with a linear number of training objectives.
- Method: A shared latent space and cross-attention between diffusion models enable simultaneous generation of arbitrary output groups without training on every generation combination.
- Results: Across eight multimodal datasets, CoDi shows strong synthesis quality, matching or exceeding single-to-single modality state of the art in reported examples.
2 Related Works
Related work covers diffusion-based generation and multimodal representation learning, while Figure 2 summarizes CoDi’s alignment-based architecture. These lines of work motivate shared representations and cross-modal generation.
- Diffusion Models: Diffusion models generate data by progressively denoising representations, with latent diffusion reducing modeling dimensionality through latent-space encoding.
- CoDi Architecture: Figure 2 depicts Bridging Alignment for aligned prompt encoders, Latent Alignment for diffusion-model interaction, and a linear number of training objectives.
- Multimodal Modeling: Multimodal modeling seeks uniform representations across modalities for cross-modal understanding and supports tasks including vision-language, video-audio, and video-speech-language applications.
3 Methodology
CoDi builds modality-specific diffusion models and aligns their prompt and latent spaces so arbitrary input combinations can condition generation of arbitrary output combinations. Composable cross-modal attention and bridging alignment support joint generation, including unseen modality combinations.
- Preliminary: Latent Diffusion Model: Latent diffusion models encode each modality into a lower-dimensional latent space, then learn to denoise those representations for generation.The framework uses modality-specific LDMs for text, image, video, and audio.
- Composable Multimodal Conditioning: CoDi conditions generation on arbitrary mixtures of text, image, video, and audio by aligning their prompt encoders and interpolating the resulting representations.Weighted interpolation enables zero-shot multi-conditioning from models trained with single conditioning.
- Composable Multimodal Conditioning: Bridging Alignment uses text as a shared modality to align conditional encoders efficiently when direct paired data between some modality pairs is scarce or unavailable.This avoids the computational burden of optimizing all modality pairs simultaneously.
- Composable Diffusion Design: CoDi independently trains modality-specific LDMs and later integrates them, preserving single-modality generation quality while supporting image, video, audio, and text synthesis.The implementation includes separate image, video, audio, and text diffusion models, with temporal modules extending the image diffuser for video.
- Joint Multimodal Generation by Latent Alignment: Latent Alignment projects one modality's noisy latent into a shared space, where another modality's UNet applies cross-attention during joint diffusion.The cross-attention weights and modality-specific environment encoders are trained with paired multimodal data while diffuser parameters are selectively frozen.
- Joint Multimodal Generation by Latent Alignment: Training a linear number of joint-generation tasks enables polynomially many modality combinations, including combinations unseen during training.Although trained on Text+Audio, Text+Image, and Video+Audio joint tasks, CoDi generates combinations such as joint image-text-audio.
4 Experiments
CoDi is trained through modality-specific synthesis, multimodal alignment, and joint-generation objectives across diverse image, text, audio, video, and audiovisual datasets.
- Training Objectives and Datasets: Training covers single-modality synthesis, joint multimodal generation, and contrastive learning for prompt-encoder alignment.The datasets span image-text, audio-text, audio-video, and video-text domains.
- Training Objectives and Datasets: Laion400M supports text→image, image→text, and joint image-text generation tasks.For joint generation, truncated captions condition generation of the original caption and image.
- Evaluation Setup: FID, MSR-VTT performance, audio-generation comparisons, and captioning scores provide modality-specific evaluation benchmarks.The supplied table captions identify COCO-caption, MSR-VTT, and AudioCaps evaluation settings.
- Training Objectives and Datasets: Audio training combines Freesound 500K, AudioSet, and AudioCaps for text-audio generation and related paired tasks.Audio examples are clipped into 10-second segments for training.
- Training Objectives and Datasets: Video training uses WebVid for text-to-video and video-text contrastive learning, plus HD-Villa-100M for image-to-video generation.The middle video frame serves as the input image for image-to-video training.
- Training Objectives and Datasets: Audio-video generation uses sound-oriented AudioSet and SoundNet resources, including image-conditioned joint audio-video tasks.The middle frame of the target video is used as the prompt image.
5 Evaluation Results
CoDi is evaluated on single-modality, multi-condition, and joint multi-output generation. Results show competitive unimodal synthesis, flexible conditioning on multiple inputs, and stronger coherence for jointly generated outputs.
- Evaluation Overview: Evaluation covers single-modality generation, multi-condition generation, and multi-output joint generation with quantitative benchmarks and visual demonstrations.The experiments assess generation quality across these three settings.
- Multi-Condition Generation: Multi-condition demonstrations include text+audio→image, text+audio→video, and video+audio→text.These examples illustrate generation from multiple input modalities.
- Single Modality Generation: CoDi achieves state-of-the-art audio captioning and audio generation, while remaining competitive for image and video generation.It also matches autoregressive transformer-based state of the art on image captioning and is the first diffusion model reported for video captioning.
- Joint Multi-Output Generation: Joint outputs include text→video+audio, text→image+text+audio, and text+audio+image→video+audio.The demonstrations cover simultaneous generation of multiple output modalities.
- Multi-Condition Generation: CoDi generates high-quality images and videos from assorted input combinations, including combinations absent from its multi-condition training data.Image quality is evaluated with FID, while video evaluation measures similarity to ground-truth text.
- Joint Multi-Output Generation: Joint generation consistently produces higher SIM scores than independent generation across tested modality combinations.SIM measures cross-modal embedding similarity using aligned encoders; higher values indicate more aligned generated modalities.
6 Conclusion
The paper concludes that CoDi processes and simultaneously generates text, image, video, and audio across varied input combinations, producing coherent multimodal outputs.
- Conclusion: CoDi processes and simultaneously generates modalities across text, image, video, and audio.The conclusion describes flexible generation of single or multiple modalities from assorted inputs.
- Conclusion: The experiments establish a foundation for further work on generative AI and more holistic human-computer interactions.This consequence is stated within the paper’s conclusion.
A Model Architecture and Configuration
CoDi uses modality-specific diffusion architectures, with a shared diffuser for image and video generation and additional temporal mechanisms for video.
- Model Architecture: Each modality-specific diffuser is based on a UNet architecture with modality-dependent variations.The architecture configuration is summarized in Table 11.
- Model Architecture: The video architecture adds temporal attention and temporal shift, while image and video generation use the same diffuser.These are the notable architecture distinctions identified in the supplied passages.
A.2 Video LDM Architecture
The video LDM extends a base image UNet with temporal mechanisms to model video sequences. It uses temporal attention and temporal shift around residual blocks.
- Temporal attention is added before each residual block to perform self-attention across video time steps.The height and width dimensions are flattened into the batch dimension before attention operates on time.
- Temporal shift is also inserted before each residual block in the video architecture.Channels are split into k chunks and shifted along the temporal dimension.
- The architecture is based on an image UNet augmented with temporal attention and temporal shift.
B Model Training
Model training combines prompt-encoder alignment, diffusion objectives, video-specific curriculum learning, and joint-generation optimization. The procedures use Adam with stage-specific hyperparameters.
- Prompt Encoders Training: Prompt encoders are trained with bridging alignment using contrastive learning between all prompt encoders.The optimizer learning rate is 1e-4 with weight decay 1e-4.
- Diffusion Model Training: Diffusion models are trained using objectives and hyperparameters specified in the paper’s training tables.The video LDM additionally uses a specific training curriculum.
- Diffusion Model Training: Video training progressively increases resolution and frame rate while changing from text-conditioned to image-conditioned video generation.Training proceeds from WebVid at 256-frame resolution and 4 FPS to HDVILLA and ACAV at 512-frame resolution and 8 FPS.
- Joint Generation Training: Joint generation is trained by aligning environment encoders and optimizing only cross-attention layers in the diffusion models.This stage uses Adam with learning rate 1e-5 and weight decay 1e-4.
C Training Datasets
The training data includes broad video collections, audiovisual resources, and datasets used for video and audiovisual learning. These resources support modality-specific and multimodal training tasks.
- The section introduces additional details about the video and audiovisual training datasets.
- Video: WebVid contains over 1.2 million silent video clips spanning diverse categories, while HD-Villa-100M contains over 100 million YouTube video clips.WebVid supports text→video and video-text contrastive learning; HD-Villa-100M provides high-quality videos of at least 720P.
- Audiovisual: SoundNet originally contains over two million sounds, and the authors collected all currently accessible 1M videos.The sound categories include music, animal sounds, natural sounds, and environmental sounds.
D Limitations & Broader Impacts
The paper discusses broader risks associated with deploying CoDi, especially misuse of realistic synchronized outputs and the reflection of bias in generated content. These concerns include deepfakes, misinformation, fraud, and stereotyping.
- Deepfakes and Misinformation: CoDi’s realistic and synchronized multimodal outputs raise concerns about creating and disseminating deepfakes.
- Deepfakes and Misinformation: Malicious actors could use fabricated videos or audio clips for misinformation, fraud, or other harmful purposes.
- Bias and Stereotyping: Biased or stereotyped training data may cause generated multimodal outputs to reflect those biases or stereotypes.