LightCache: Memory-Efficient, Training-Free Acceleration for Video Generation

1 INTRODUCTION

• Cached acceleration can trigger GPU out-of-memory errors when generating a single sample, because inference offers no natural workload distribution across GPUs.The authors report encountering this with DeepCache on EasyAnimate using four 45 GB L40 GPUs.Source: §1 · p. 3
• Encoding memory remains consistently low, while denoising or decoding can determine peak usage, so the authors focus memory optimization on those latter stages.Reducing denoising memory alone may not lower peak memory when decoding is the peak stage.Source: §1 · p. 3
• Feature chunking divides feature-map height and width by layer to reduce denoising memory while maintaining generation quality.Different layers can be chunked to different degrees.Source: §1 · p. 3
• Asynchronous swapping moves inactive cached feature maps to CPU and overlaps transfers with GPU computation to mitigate latency overhead.The approach follows the idea of moving inactive components off the GPU.Source: §1 · p. 3
• Sequentially decoding smaller batches through VAE slicing applies even to a single video because classifier-free guidance doubles feature maps across frames.The passage describes conditional and unconditional maps across a total of 2 × N frames for N generated frames.Source: §1 · p. 3

2.1 PRELIMINARY

• Diffusion progressively adds scheduled Gaussian noise and trains a network to reverse the process, while U-Net combines down-sampled features with up-sampled features through skip connections.The paper describes down-sampling as encoding and up-sampling as decoding.Source: §2.1 · pp. 3–4

2.2 ASYNCHRONOUS SWAPPING

• DeepCache reuses slowly evolving feature maps across timesteps; LightCache stores cached features on the CPU and retrieves them when needed, slightly increasing inference time while reducing GPU memory.A layer’s feature map is recalculated once every n steps and reused in intervening steps.Source: §2.2 · pp. 4–5

2.3 FEATURE CHUNK AND SLICED DECODING

• Feature chunking divides spatial height and width while keeping batch, frames, and channels unchanged; larger chunks reduce memory but may degrade quality.The paper reports that chunking works only during denoising, so it cannot lower peak memory when decoding uses more memory.Source: §2.3 · p. 5
• Table 1 reports that LightCache preserves quality, accelerates inference, and reduces memory below the baseline, unlike DeepCache and FME.The caption says DeepCache increases memory usage, while FME decreases speed-up; both may cause quality degradation across models.Source: Table 1 · p. 6
• For decoding, LightCache processes video frames individually with VAE slicing, then merges the decoded images into a video.This turns one computation into multiple computations; the paper says decoding time is negligible under DeepCache acceleration.Source: §2.3 · p. 6

3.1 ANALYSIS

• Experiments used four NVIDIA L40S GPUs, batch size 1, and 512×512 resolution except for SVD at 1024×576; inference steps varied by model.DeepCache on EasyAnimate ran out of memory, and FME was not applicable to DiT architectures.Source: §3.1 · pp. 6–7
• LightCache offers the most favorable speed–quality–memory balance: faster than baseline, lower peak memory than baseline and DeepCache, and quality close to DeepCache.The authors report that FME reduces memory more, but at the cost of substantial quality loss.Source: §3.1 · p. 7

3.2 ABLATION STUDIES

• Removing swapping raised peak memory across all three stages, removing slicing raised decoding memory, and removing chunking raised denoising memory.The ablation compares memory consumption across encoding, denoising, and decoding stages.Source: Table 2 · p. 7, §3.2 · p. 8
• The method was not constrained to one scheduler: Euler, DDIM, and PNDM were tested, with default Euler best on time cost, LPIPS, PSNR, and SSIM.Table 3 covers samplers on SVDI2V-XT and marks the official default sampler.Source: §3.2 · p. 8, Table 3 · p. 8

3.3 VISUALIZATION

• Framewise violin plots show LightCache’s quality consistency with DeepCache, while FME produces unstable outcomes; LPIPS better aligns with human perception than PSNR or SSIM.PSNR and SSIM reflect low-level pixel and structural similarity, respectively.Source: §3.3 · p. 8

4.1 TRAINING-FREE ACCELERATION OF DIFFUSION MODELS.

• Training-free diffusion acceleration exploits structural redundancy through inference skipping and computation-graph optimization, alongside quantization that reduces memory and accelerates computation without retraining.The cited approaches include Ma et al. (2024b), Tian et al. (2025), Yu et al. (2023), and Zhou et al. (2025) for inference skipping, and Siddegowda et al. (2022), Wang et al. (2025), Zhao et al. (2025; 2024a), and Shang et al. (2023) for quantization.Source: §4.1 · pp. 8–9

4.2 CACHE IN DIFFUSION MODELS.

• Cache-based diffusion methods reuse earlier features or noise states to avoid redundant denoising computations, including Ma et al. (2024b), Wimbauer et al. (2024), Zhou et al. (2025), and Agarwal et al. (2024).Other examples include PAB, which reuses stable-phase attention features, and FORA, which caches Attention and MLP features at fixed intervals.Source: §4.2 · p. 9

5 CONCLUSION

• LightCache combines asynchronous swapping, feature chunking, and VAE slicing to reduce peak memory while keeping added time overhead below its acceleration gains.The framework is training-free and compares favorably with the baseline, DeepCache, and FME on peak memory and speed while maintaining quality close to DeepCache.Source: §5 · p. 9
• Future work will explore combining LightCache with complementary training-free methods and extending it to DiT models, longer video sequences, and multimodal generation.The stated motivation for these extensions is the importance of memory efficiency and inference speed in those settings.Source: §5 · p. 10

6 ETHICS STATEMENT

• The authors report no human subjects, personally identifiable data, or sensitive information, and state that the work raises no ethical concerns.Source: §6 · p. 10

• The authors make the code available in an anonymous project.Source: §7 · p. 10
• The code file contains detailed hyperparameters, inference procedures, and data processing steps.Source: §7 · p. 10
• Random seeds are fixed, and results were verified across multiple independent runs.Source: §7 · p. 10

• LLMs assisted with refining the manuscript’s clarity, grammar, and style.Source: §A.1 · p. 13
• LLM use was limited to language polishing, not the generation of substantive ideas, analyses, or results.Source: §A.1 · p. 13
• The authors state that all scientific content, interpretations, and conclusions are their own.Source: §A.1 · p. 13