SimDA: Simple Diffusion Adapter for Efficient Video Generation

1. Introduction

• Prior T2V approaches that adapt T2I models accelerate convergence with temporal modules, but can involve 4B or even 16B parameters.Other existing approaches often train from scratch, which can be time-consuming on complex video data.Source: §1 · p. 2
• SimDA freezes Stable Diffusion and fine-tunes only newly added modules, adding 0.02% parameters relative to the T2I model.Its Latent-Shift Attention replaces spatial attention to improve temporal modeling and consistency without adding parameters.Source: §1 · p. 2
• The paper presents tuning extremely few parameters as a way to obtain good video-generation results.Its lightweight spatial and temporal adapters are intended to model temporal relations with negligible cost.Source: §1 · p. 2
• SimDA extends its adapter approach to video super-resolution and text-guided video editing.The super-resolution model generates 1024 × 1024 videos; editing training is reported as 3× faster while retaining comparable results.Source: §1 · p. 2
• SimDA alleviates training cost and speeds inference while remaining competitive with other methods.The authors also report training below 8GB GPU memory and inference 39× faster than CogVideo [39].Source: §1 · p. 2

Text-to-Video Generation

• Open-wild T2V development is limited by video modeling challenges and the need for large text-video datasets; prior work includes GAN-based, pretrained-model, and joint image-video approaches.Reported systems can be large: Make-A-Video [82] uses six models and 9.7B parameters, while Imagen Video [35] uses eight models and 16.3B parameters.Source: §2 · pp. 2–3

Text guided Video Editing

• Text-guided video editing manipulates existing videos using textual input, with approaches including single-video fine-tuning, attention-map editing, and layer-wise editing.Tune-A-Video [100] fine-tunes a T2I model on one video to generate new videos with similar motion patterns.Source: §2 · p. 3

Parameter-Efficient Transfer Learning

• Parameter-efficient transfer learning reduces trainable parameters to lower computation costs, while SimDA applies it to T2V rather than classification or detection.Prior approaches in computer vision mainly focus on adapting models for simple classification or detection tasks.Source: §2 · p. 3

Temporal Shift Module

• Temporal-shift methods move channels or patches across frames; SimDA's LSA shifts patches and uses all current-frame tokens as keys and values for temporal consistency.TPS's direct patch shifting can lead to inconsistency in generation.Source: §2 · p. 3

3.1. Preliminaries of Stable Diffusion

• The method starts from Stable Diffusion’s latent diffusion process, encoding images into latent features, adding noise, and training a U-Net to predict that noise from text-conditioned latents.At inference, denoising starts from Gaussian noise and the pretrained decoder reconstructs the output.Source: §3.1 · p. 3

3.2. Pipeline

• For video, SimDA encodes each frame, diffuses the latents, and uses an inflated U-Net with a 3D convolution block, temporal adapter, latent-shift attention, and spatial adapters before decoding the denoised latents.The pipeline uses DDIM sampling at inference.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Modeling

• Spatial adapters use two fully connected layers with an intermediate activation, while the temporal adapter uses depth-wise 3D convolution on lower-dimensional inputs, alleviating temporal-modeling complexity and achieving efficient training memory use and fastest inference among competitive approaches.Source: §3.3 · pp. 4–5
• Latent-Shift Attention shifts patch-level tokens from preceding frames onto the current frame and concatenates them for keys and values, reducing attention complexity to O(2LN^2) from O(L^2N^2).The authors state that it learns relationships between adjacent frames and supports better temporal consistency.Source: §3.3 · p. 5

3.4. Super Resolution and Editing Models

• The super-resolution model conditions on low-resolution video concatenated frame by frame with noisy latents, uses spatial and temporal adapters, and fine-tunes only the newly added modules.Its architecture is similar to the first-stage T2V model.Source: §3.4 · pp. 5–6
• The framework also supports one-shot, text-guided video editing by sampling a novel video from latent noise inverted from the input video and guiding it with an edited prompt.Figure 4 illustrates this inference process.Source: §3.4 · p. 6, Figure 4 · p. 6

4.1. Implementation Details

• The T2V system uses a 256×256 frame-generation model followed by a 4× upsampler that produces 1024×1024 video.The general T2V model is trained on WebVid-10M.Source: §4.1 · p. 6

4.2. Evaluation on Text-to-Video Generation

• 0.2945 average CLIPSIM on zero-shot MSR-VTT surpasses most competitors, while Make-A-Video and VideoFactory score higher using additional HD-VILA data.The evaluation generates 2,990 videos from randomly selected MSR-VTT prompts.Source: §4.2 · p. 7
• 363.98 FVD and 0.3054 CLIPSIM on WebVid validation; the paper reports these scores exceed VideoFusion and LVDM and are competitive with VideoFactory.The evaluation uses 4,476 WebVid text-video pairs excluded from training.Source: §4.2 · p. 7, Table 3 · p. 7
• SimDA is significantly smaller than prior methods and has faster inference than the compared methods, while achieving superior performance across benchmarks despite fewer parameters.The speed comparison includes CogVideo [39], Latent-Shift [1], and LVDM [32].Source: §4.2 · p. 9

4.3. Evaluation on Text-guided Video Editing

• SimDA received the most user-study votes for both temporal consistency and textual match in video editing.The study evaluated 75 edited videos per model across object, background, and style edits.Source: §4.3 · p. 9

4.4. Ablation Study

• Without the Temporal Adapter, FVD rises and generated sequences become chaotic.The ablations use 1K WebVid validation samples.Source: §4.4 · p. 9
• Without the Attention Adapter and FFN Adapter, FVD and CLIPSIM worsen; without the Spatial Adapter, generated videos show prompt-understanding misconceptions.Source: §4.4 · p. 9, Table 6 · p. 10
• Replacing Latent-Shift Attention with single-frame spatial attention lowers FVD and CLIPSIM, and within-video CLIPSIM falls from 96.4 to 94.5.The authors attribute more consistent videos to LSA modeling adjacent-frame relationships.Source: §4.4 · p. 10, Table 6 · p. 10

• SimDA is a parameter-efficient video diffusion model for text-guided video generation and editing, using lightweight spatial and temporal adapters.The paper presents it as an efficient T2V fine-tuning baseline.Source: §5 · p. 10