Source-linked AI summary
WorldGPT: A Sora-Inspired Video AI Agent as Rich World Models from Text and Image Inputs
Deshun Yang, Luhui Hu, Yu Tian, Zihao Li, Chris Kelly, Bang Yang, Cindy Yang, Yuexian Zou
TL;DR
Text-to-video generation still faces challenges in temporal consistency and action smoothness. WorldGPT pairs prompt enhancement with diffusion-based video generation, and reports gains over comparison models in alignment, motion, and temporal consistency, alongside weaker frame quality.
Problem
Maintaining temporal consistency and action smoothness remains a challenge for text-to-video diffusion models.
Source: §1 · p. 1Method
WorldGPT uses ChatGPT to turn input text into task-specific prompts and combines visual and language models in a multimodal video-generation pipeline.
Source: §3.1 · p. 4, §3.2 · p. 5, §3.3 · p. 5Results
WorldGPT outperforms DynamiCrafter and I2VGen-XL in control-video alignment, motion effects, and temporal consistency, while performing relatively worse in frame quality; human evaluation also reports over 60% preference for its motion quality and text-video alignment.
Source: §4.2 · pp. 7, 9Takeaways & Limitations
The conclusion presents prompt-based controllability as allowing users to tailor video-generation results to their needs and preferences.
Source: §5 · pp. 9–10WorldGPT performs relatively worse in frame quality than DynamiCrafter and I2VGen-XL, which the authors suggest may reflect key-frame image quality.
Source: §4.2 · p. 7
Abstract
from arXiv · showhide
Several text-to-video diffusion models have demonstrated commendable capabilities in synthesizing high-quality video content. However, it remains a formidable challenge pertaining to maintaining temporal consistency and ensuring action smoothness throughout the generated sequences. In this paper, we present an innovative video generation AI agent that harnesses the power of Sora-inspired multimodal learning to build skilled world models framework based on textual prompts and accompanying images. The framework includes two parts: prompt enhancer and full video translation. The first part employs the capabilities of ChatGPT to meticulously distill and proactively construct precise prompts for each subsequent step, thereby guaranteeing the utmost accuracy in prompt communication and accurate execution in following model operations. The second part employ compatible with existing advanced diffusion techniques to expansively generate and refine the key frame at the conclusion of a video. Then we can expertly harness the power of leading and trailing key frames to craft videos with enhanced temporal consistency and action smoothness. The experimental results confirm that our method has strong effectiveness and novelty in constructing world models from text and image inputs over the other methods.
1 Introduction
Text-to-image systems established strong prompt-conditioned image generation, while text-to-video models must interpret language and generate sequential frames. WorldGPT proposes combining diffusion methods with multimodal processing to address temporal-spatial consistency, diversity, and video evaluation for generation from text and images.
1 Introduction
- Text-to-image systems such as DALL-E 2, CogView 2, Imagen, and Stable Diffusion generate high-quality images from text, while text-to-video systems must produce sequential frames.DALL-E 2 combines a transformer-based language model with StyleGAN 2, and Stable Diffusion uses CLIP text conditioning for latent image representations.Source: §1 · pp. 1–2
- WorldGPT targets temporal-spatial consistency, diverse generation, and video inference evaluation by applying multimodal processing to text-and-image video generation.The proposed methods incorporate dynamic scene modeling, spatiotemporal prediction, and multimodal fusion.Source: §1 · p. 2
2 Related Work
Related work covers language models, text-to-image generation, image-to-video synthesis, and Stable Diffusion. It describes model objectives and conditioning mechanisms relevant to the paper’s generation pipeline.
2.1 Large Language Model
- Large language models handle natural-language tasks, and the cited objective minimizes the negative log probability of each token given preceding tokens and model parameters.The passage identifies preceding tokens as context and model parameters as θ.Source: §2.1 · p. 2
2.2 Text-to-Image Generation
- Text-to-image synthesis converts natural-language descriptions into images, with DALL-E cited as a system that creates realistic images from prompts.The discussion also notes the development of conditional GANs for paired image data.Source: §2.2 · pp. 2–3
2.3 Image-to-Video Generation
- DynamiCrafter is presented as an image-to-video method that synthesizes a sequence from an input image using motion flow and appearance-change fields.Its loss compares synthesized frames with target frames, with minimization described as ensuring temporal and spatial consistency.Source: §2.3 · p. 3
2.4 Stable Diffusion
- Stable Diffusion iteratively denoises an image, using a U-Net and text conditioning through CLIP, and is described as generating high-quality images in a low-dimensional latent space.The diffusion process is represented as a Markov chain with a noise term and a coefficient between zero and one.Source: §2.4 · p. 3
3 WorldGPT
WorldGPT combines a ChatGPT prompt enhancer with full video translation, integrating leading models to generate videos from image-text inputs without pre-training. Its pipeline turns task-specific prompts into a key frame and then a video, with the framework described as improving semantic coherence between visual content and text.
3.1 Framework Overview
- The framework combines a Prompt Enhancer and Full Video Translation, integrating leading models for zero-shot video generation from image-text inputs without pre-training.The translation component contains Key Frame Generator and Video Generator subsystems.Source: §3.1 · p. 4
3.2 Prompt Enhancer
- ChatGPT refines an input sequence into task-specific prompts to improve prompt communication and subsequent model execution.The prompt enhancer is described as helping ChatGPT grasp user intentions and context when formulating prompts.Source: §3.2 · pp. 4–5
3.3 Key Frame Generator
- GroundingDino detects objects and creates target masks, while ChatGPT-derived prompts guide Stable Diffusion in generating the video’s end frame.The approach is described as maintaining thematic consistency with the original image while offering creative flexibility.Source: §3.3 · p. 5
3.4 Video Generator
- DynamiCrafter generates a frame sequence from the starting and ending frames, after which ChatGPT-generated background details and requirements refine the video.The refinement is described as retaining the generated structure while incorporating the client’s vision for background elements and storytelling.Source: §3.4 · pp. 5–6
4 Experimental Results
WorldGPT was evaluated in zero-shot image-text-to-video experiments against other models, with reported advantages in alignment, motion, temporal consistency, and controllability, but weaker frame quality. Human evaluation found comparable visual quality and a preference for WorldGPT on motion quality and text-video alignment.
4.1 Implementation Details
- The evaluation used one NVIDIA 4090 GPU, zero-shot inference without task-specific fine-tuning or training, and 256-scale models for comparative evaluation.The experiments included quantitative evaluation of coherence, text alignment, and video quality, plus qualitative controllability assessment.Source: §4.1 · p. 6
4.2 Experimental results and analysis
- On AIGCBench, WorldGPT outperforms DynamiCrafter and I2VGen-XL in control-video alignment, motion effects, and temporal consistency, but performs relatively worse in frame quality.Table 1 reports quantitative comparisons for zero-shot generation.Source: §4.2 · pp. 6–7, Table 1 · p. 7
- Visualized comparisons with DynamiCrafter show greater controllability and flexibility for WorldGPT when generation is controlled through prompts.The qualitative evaluation used customized real-world datasets and images from AIGCBench.Source: §4.2 · pp. 7–8
- In the Fig. 2 examples, WorldGPT better reflects textual inputs and distinguishes related prompts, especially when prompts describe targets absent from the source image.For simple prompts, the authors note that alignment with the original image may still be imperfect.Source: Figure 2 · p. 8, §4.2 · p. 9
- In human comparisons across 50 AIGCBench image-text sets, over 60% of participants chose the evaluated algorithm for motion quality and text-video alignment; its visual quality was comparable to DynamiCrafter and I2VGen-XL.The evaluation assessed text-video alignment, motion quality, and visual quality.Source: §4.2 · p. 9, Table 2 · p. 9
5 Conclusion
WorldGPT combines text semantics with visual interpretation in a video-generation pipeline and emphasizes prompt-based user control. Users can shape generation and tailor results to their needs and preferences.
5 Conclusion
- WorldGPT combines high-level semantic understanding from text with advanced visual interpretation in a video-generation pipeline.The conclusion describes this combination as producing videos that encapsulate rich and realistic world models.Source: §5 · p. 9
- Its image-text algorithm lets users manipulate and shape video generation through prompts.The paper identifies controllability as a distinguishing feature of its approach.Source: §5 · pp. 9–10
- Prompt-based control lets users customize and tailor generated results to their needs and preferences.The conclusion frames the system as combining language and vision with user controllability.Source: §5 · p. 10