PaperlayerView exact revision on arXiv

Source-linked AI summary

FlashMotion: Few-Step Controllable Video Generation with Trajectory Guidance

Quanhao Li, Zhen Xing, Rui Wang, Haidong Cao, Qi Dai, Daoguo Dong, Zuxuan Wu

arXiv:2603.12146v1cs.CVcs.AIcs.LGcs.MM

TL;DR

Trajectory-controllable video generation remains costly because existing methods use multi-step denoising, while directly distilling them can reduce quality and trajectory accuracy. FlashMotion trains and adapts a few-step generator through staged diffusion and adversarial objectives, and reports superior visual quality and trajectory consistency against existing few-step and multi-step methods.

  • Problem

    Existing trajectory-controllable video generation requires multi-step denoising, while direct distillation can degrade video quality and trajectory accuracy.

  • Method

    FlashMotion trains a trajectory adapter on a multi-step generator, distills the generator to few steps, and finetunes the adapter with hybrid diffusion and adversarial objectives.

  • Results

    FlashMotion achieves superior performance to few-step distillation methods and multi-step trajectory-guided video generation methods in visual quality and trajectory consistency.

  • Takeaways & Limitations

    FlashMotion provides few-step trajectory-controllable video generation while evaluating long-sequence quality and trajectory accuracy with FlashBench.

Abstract

from arXiv · show

Recent advances in trajectory-controllable video generation have achieved remarkable progress. Previous methods mainly use adapter-based architectures for precise motion control along predefined trajectories. However, all these methods rely on a multi-step denoising process, leading to substantial time redundancy and computational overhead. While existing video distillation methods successfully distill multi-step generators into few-step, directly applying these approaches to trajectory-controllable video generation results in noticeable degradation in both video quality and trajectory accuracy. To bridge this gap, we introduce FlashMotion, a novel training framework designed for few-step trajectory-controllable video generation. We first train a trajectory adapter on a multi-step video generator for precise trajectory control. Then, we distill the generator into a few-step version to accelerate video generation. Finally, we finetune the adapter using a hybrid strategy that combines diffusion and adversarial objectives, aligning it with the few-step generator to produce high-quality, trajectory-accurate videos. For evaluation, we introduce FlashBench, a benchmark for long-sequence trajectory-controllable video generation that measures both video quality and trajectory accuracy across varying numbers of foreground objects. Experiments on two adapter architectures show that FlashMotion surpasses existing video distillation methods and previous multi-step models in both visual quality and trajectory consistency.

1 Introduction

FlashMotion addresses the difficulty of accelerating trajectory-controllable video generation without sacrificing visual quality or trajectory accuracy. It combines staged generator and adapter training with FlashBench, a long-sequence benchmark for evaluating both dimensions.

  • Existing trajectory-controllable video methods rely on multiple denoising steps, creating computational overhead and leaving few-step acceleration largely unexplored.
  • Directly applying video distillation to trajectory-controllable generation can degrade both video quality and trajectory accuracy because the original adapter is incompatible with the few-step generator.
  • Hybrid diffusion and adversarial training aligns the adapter with the few-step generator, combining pixel-level trajectory supervision with distribution-level video consistency.
  • FlashBench evaluates long video sequences across six foreground-object categories using both visual quality and trajectory-control accuracy.
  • FlashMotion uses a three-stage framework: train a trajectory adapter with a multi-step generator, distill the generator, then finetune the adapter for the few-step model.
  • FlashMotion is reported to outperform few-step distillation methods and multi-step trajectory-guided video generation methods.

2 Related Works

Related work spans trajectory-control methods and video diffusion distillation. Existing approaches provide motion control or efficient synthesis, but distillation methods are not specifically designed for trajectory-controllable generation.

  • Trajectory-controllable video methods control foreground-object motion through training-free attention manipulation or explicit trajectory supervision.
  • Video diffusion distillation methods include consistency, score, and adversarial distillation, each targeting efficient few-step synthesis through a different training objective.
  • Existing video distillation methods can achieve efficiency gains but often degrade visual quality and trajectory accuracy when applied to trajectory-controllable generation.

3 Method

FlashMotion combines adapter training, generator distillation, and hybrid adapter finetuning to enable few-step trajectory-controllable video generation. Its architecture uses trajectory-feature injection and a diffusion discriminator to preserve motion control while improving generated-video quality.

  • Overview: FlashMotion uses three stages: train SlowAdapter on SlowGenerator, distill FastGenerator, then finetune the adapter with diffusion and adversarial losses.The final stage aligns the adapter with the few-step generator.
  • Trajectory Adapter: The trajectory adapter injects trajectory features into corresponding DiT blocks through zero-initialized convolutions to guide video generation.Two adapter architectures are evaluated: ControlNet-based and lightweight ResNet-based designs.
  • Fast Generator: DMD distills SlowGenerator into FastGenerator by aligning teacher and student video distributions through Kullback–Leibler divergence.The student maps Gaussian noise to clean video samples, while real and fake score models support distribution matching.
  • Fast Adapter: Hybrid finetuning combines diffusion and adversarial objectives because directly transferring SlowAdapter to FastGenerator degrades visual quality and trajectory accuracy.The adapter is initialized from SlowAdapter and optimized in an alternating scheme with a diffusion discriminator.
  • Adversarial Training: The diffusion discriminator distinguishes real and generated videos using selected DiT features and attention-based classifiers.Its backbone is initialized from Wan2.2-TI2V-5B, while classifier components produce the final real-versus-fake logits.
  • FlashBench: FlashBench evaluates long-sequence trajectory-controllable generation because prior benchmarks focus mainly on short videos.The benchmark addresses long-term temporal consistency and trajectory controllability, including videos up to 121 frames.

4 Experiment

The experiments evaluate FlashMotion across long-sequence benchmarks, visual-quality and trajectory-accuracy metrics, qualitative comparisons, and component ablations. FlashMotion outperforms prior multi-step and few-step approaches, while ablations show that adapter alignment and hybrid objectives are important.

  • Experimental settings: FlashMotion is evaluated on FlashBench, MagicBench, and DAVIS using FID, FVD, Mask_IoU, and Box_IoU for ResNet and ControlNet adapters.Denoising time is measured for generating 121 frames on one A100 GPU.
  • Comparison setup: FlashMotion is compared with trajectory-controllable methods and few-step distillation baselines using both visual quality and trajectory accuracy.The comparison includes MagicMotion, Tora, DragAnything, SGI2V, LeviTor, Wan2.2-TI2V-5B with SlowAdapter, and adapted distillation baselines.
  • Comparison results: FlashMotion outperforms previous multi-step trajectory-controllable methods and few-step distillation baselines in qualitative comparisons.Figure 4 compares the input image, prompt, trajectory, and generated videos across these methods.
  • Ablation studies: Removing FastAdapter training degrades both visual quality and trajectory accuracy and can cause severe color shifts.The ablation indicates that SlowAdapter cannot directly control FastGenerator.
  • Ablation studies: Removing diffusion loss lowers trajectory accuracy and visual quality, while removing GAN loss slightly improves trajectory accuracy but reduces visual quality by nearly 90%.The GAN-loss ablation introduces severe blurring artifacts.
  • Ablation studies: Dynamic loss scaling improves visual quality, and the full discriminator architecture achieves the best performance across all metrics.Semantic Self-Attention improves visual quality, while Trajectory Cross-Attention strengthens trajectory control accuracy.

5 Conclusion

FlashMotion enables few-step trajectory-controllable video generation through a three-stage training paradigm and introduces FlashBench for long-sequence evaluation. Extensive experiments show superior visual fidelity and trajectory consistency against few-step distillation and prior multi-step methods.

  • Conclusion: FlashMotion trains a trajectory adapter, distills the multi-step generator into a few-step version, and finetunes the adapter with diffusion and adversarial objectives.The three stages align trajectory control with the few-step generator.
  • Conclusion: FlashBench evaluates long-sequence trajectory-controllable video generation using visual quality and trajectory accuracy across foreground-object categories.The benchmark complements the training framework’s evaluation of both fidelity and controllability.
  • Conclusion: FlashMotion surpasses existing few-step distillation approaches and prior multi-step trajectory-controllable models in visual fidelity and trajectory consistency.The conclusion reports this pattern across the evaluated methods.

6 Additional Ablation results

The ablations show that every major FlashMotion component contributes to preserving video quality and trajectory accuracy. The full discriminator and complete training strategy provide the strongest overall behavior.

  • Fast Adapter: Removing FastAdapter training consistently reduces video quality and trajectory accuracy across all three benchmarks.This supports adapting the trajectory adapter to the few-step generator rather than directly reusing SlowAdapter.
  • Diffusion Loss: Removing diffusion loss decreases trajectory alignment for both adapter architectures and also degrades image and video quality.The ablation identifies diffusion supervision as important for maintaining consistency with user-specified trajectories.
  • GAN Loss: Removing GAN loss slightly improves trajectory accuracy but reduces image and video quality by approximately 90%, causing severe blurring artifacts.The result highlights a quality–trajectory trade-off when adversarial supervision is omitted.
  • Dynamic Diffusion Loss Scaling: Disabling dynamic diffusion loss scaling lowers image and video quality across all three benchmarks and produces noticeable blurring artifacts.The comparison fixes the loss scale to 1 during training.
  • Discriminator Architecture: The full discriminator architecture achieves the best results across all evaluation metrics and benchmarks.Semantic Self-Attention improves visual realism, while Trajectory Cross-Attention strengthens trajectory control; Video Cross-Attention alone performs worst.
  • Qualitative Ablations: Qualitative ablations show color drift and distorted object shapes without FastAdapter training, while removing diffusion loss causes objects to leave intended paths.The latter can even produce duplicated objects in the illustrated extreme case.

7 Additional Comparison results

FlashMotion is evaluated against competing methods in speed, capacity, object-count conditions, and qualitative video comparisons. It combines fast denoising and broad generation support with strong visual quality and trajectory control.

  • Model Configuration Comparison: FlashMotion provides the fastest denoising speed for both ControlNet- and ResNet-based adapters while supporting the highest resolution and longest generation length.The comparison also reports supported video length, spatial resolution, denoising latency, and parameter count.
  • Results Across Object Counts: The ControlNet variant consistently surpasses competing methods across all metrics and object-count conditions from 1–5 to more than 5 foreground objects.The comparison includes both multi-step and few-step baselines.
  • Results Across Object Counts: The ResNet-based variant achieves better visual quality than MagicMotion but remains slightly behind it in trajectory accuracy because of limited parameter capacity.This result is reported across detailed FlashBench evaluations grouped by controlled-object count.
  • Qualitative Comparisons: Across qualitative comparisons, FlashMotion accurately controls object trajectories and produces high-quality videos, while other approaches show notable artifacts and inconsistencies.The comparisons are presented in Figs. 9–15.

8 Case Studies

Case studies demonstrate FlashMotion’s adaptability across artistic styles and its ability to control camera motion and scene navigation through foreground-object bounding boxes.

  • Visual Styles: FlashMotion generates videos in dreamlike realism, surreal miniature photography, 3D cartoon rendering, and Eastern ink-wash painting.The examples use vertically oriented images to test motion consistency across challenging layouts.
  • Camera Control: Users can control zooming in and out by changing the bounding-box size of a foreground object.Examples include a cup and a woman’s mask.
  • Scene Navigation: Users can navigate bakery and museum scenes by manipulating the bounding boxes of key objects.Examples include a dinosaur, mammoth, and industrial mixer.
  • Model Capacity: FlashMotion’s model configurations support the highest resolution and longest generation length while achieving the fastest denoising speed.These configuration properties are summarized for the compared backbone architectures.

9 More Details on FlashBench

FlashBench is designed to evaluate long-sequence trajectory-controllable video generation across different numbers of foreground objects. Supplementary figures illustrate the benchmark’s video-length coverage and FlashMotion’s qualitative capabilities.

  • Benchmark Composition: FlashBench contains 600 videos grouped into six categories by foreground-object count, including 1–5 and more than 5 objects.The benchmark visualizes video-length distributions to demonstrate support for long-video evaluation.
  • Object-Count Evaluation: FlashMotion with a ControlNet-based adapter consistently outperforms competing methods across all metrics in scenes containing 4, 5, and more than 5 controlled objects.The comparison is reported in Table 8.
  • Additional Capabilities: FlashMotion supports diverse video styles, camera zoom control, and scene navigation through bounding-box manipulation.Figures 16–18 cover styles, camera movements, and navigation in bakery or museum environments.
  • Video Lengths: Figure 19 visualizes the distribution of video frame counts in FlashBench to demonstrate its support for evaluating long video generation.The figure complements the benchmark’s long-sequence design.
Loading 2603.12146v1…