PaperlayerView exact revision on arXiv

Source-linked AI summary

CineMobile: On-Device Image-to-Video Diffusion for Cinematic Camera Motion Generation

Xuyao Huang, Zelai Deng, Xu Wang, Xizhong Xiao, Zhijie Deng

arXiv:2607.03803v1cs.CVcs.AI

TL;DR

Mobile image-to-video generation with DiT models remains impractical because their large size and multi-step denoising create substantial computational overhead. CineMobile combines pruning, distillation, reinforcement learning, and hybrid quantization to deliver comparable visual quality with 40.11× faster DiT denoising than the teacher.

  • Problem

    DiT-based image-to-video models remain difficult to deploy on mobile devices because of large parameter sizes, slow generation, and limited prior mobile acceleration work.

  • Method

    CineMobile combines structured depth pruning, step distillation with reinforcement learning, and hybrid precision quantization for compact cinematic image-to-video generation.

  • Results

    40.11× faster DiT denoising and 71.89× lower DiT-FLOPs are achieved while maintaining competitive visual quality with only four denoising steps.

  • Takeaways & Limitations

    CineMobile provides a practical on-device solution for cinematic image-to-video generation across bullet time, dolly zoom, and slow motion.

Abstract

from arXiv · show

The growing demand for image-to-video creation on mobile devices has increasingly focused on cinematic motion effects like bullet time, dolly zoom, slow motion, etc. While Diffusion Transformers (DiTs) exhibit strong performance in video generation, their large parameter sizes and multi-step iterative denoising processes lead to substantial computational overhead, making efficient generation on mobile devices challenging. We propose CineMobile to bridge the gap. In particular, CineMobile adopts a three-fold optimization strategy: (1) leveraging a distillation-guided pruning approach to derive a compact yet efficient model that retains the essential video generation capabilities required for cinematic effects; (2) optimizing the compressed model into a 4-step generator via a combination of diffusion distillation and reinforcement learning; (3) employing a hybrid post-training quantization strategy to compress the model footprint to under 1 GB. Experimental results show that compared to the teacher model with the Wan 2.1 architecture, CineMobile achieves a 40x speedup in generation while maintaining comparable visual quality. Specifically, CineMobile generates 49-frame 480p videos with a per-step denoising latency of 0.6s on an NVIDIA H200 GPU and 20s on the MediaTek Dimensity 8400 Ultimate 5G platform, with a peak memory usage of 1.8 GB, demonstrating its practical applicability for mobile-based image-to-video creation.

1 Introduction

CineMobile targets mobile image-to-video generation by combining structured pruning, step distillation, and hybrid precision quantization for cinematic camera-motion effects. The approach restores the pruned model’s I2V capability, distills it into a 4-step generator, and reduces its memory footprint to under 1 GB while preserving visual quality.

  • Motivation: DiT models’ huge parameter sizes and slow generation speed limit practical mobile video generation.Wan2.1-I2V-14B requires 240 seconds to generate a 49-frame 832 × 480 video with 40 inference steps on an NVIDIA H200 GPU.
  • Method: CineMobile extends structured pruning to image-to-video DiTs while prioritizing fine portrait details and stable camera control.The method combines structured pruning with step distillation and hybrid precision quantization for mobile cinematic camera-motion generation.
  • Method: A supervised fine-tuning warm-up restores fundamental I2V generation and adapts the pruned student to the target camera-motion distribution before step distillation.The adapted AdvDMD procedure incorporates reinforcement learning into distribution-matching distillation for I2V DiTs.
  • Method: 4-bit FFN-weight quantization and 8-bit quantization of remaining components reduce CineMobile’s memory footprint to under 1 GB while preserving visual quality.The hybrid precision strategy is designed to reduce on-device memory cost.
  • Evaluation: CineMobile is evaluated on bullet time, dolly zoom, and slow motion to assess controllable and efficient on-device cinematic I2V generation.These effects represent distinct cinematic motion controls with clear motion patterns and are evaluated using high-quality portrait and motion data with VBench.

2 Related Work

Related work advances along three connected directions: increasingly capable video diffusion foundations, mobile-oriented video generation, and few-step diffusion distillation. Recent systems combine these efficiency techniques with hardware-aware model design for on-device deployment.

  • Video Diffusion Models: Video generation progressed from early video diffusion and latent video models to large-scale foundations including Stable Video Diffusion, Lumiere, CogVideoX, Open-Sora, LTX-Video, and Wan.These models target improved visual fidelity, motion coherence, and scalability.
  • On-Device Models: Recent on-device models reduce mobile video-generation costs through resolution reduction, temporal multi-scaling, pruning, adversarial fine-tuning, denoising reduction, token merging, and dynamic loading.SnapGen-V further combines an efficient spatial backbone, mobile-oriented temporal design, and adversarial low-step training for interactive generation.
  • Step Distillation: Step distillation accelerates diffusion generation by reducing denoising steps, progressing from DDIM and Progressive Distillation to Latent Consistency Models, Adversarial Diffusion Distillation, DMD, and DMD2.These methods establish the main few-step acceleration paradigms for diffusion models.
  • Step Distillation: VideoLCM, AnimateLCM, T2V-Turbo, and T2V-Turbo-v2 adapt few-step distillation to temporal video generation and achieve competitive quality with only a few sampling steps.Recent mobile systems combine step distillation with hardware-aware model design.

3 Method

CineMobile combines a shared DiT backbone with effect-specific LoRA modules and a staged compression pipeline: pruning, two-stage fine-tuning with 4-step distillation, and hybrid quantization for mobile deployment.

  • Overview: A shared backbone with effect-specific LoRA modules supports diverse cinematic camera effects while reducing deployment overhead.The shared DiT backbone is pruned first, followed by capability restoration and 4-step generator distillation.
  • 3.1 Backbone Pruning: PPCL-based pruning detects redundant contiguous layer intervals using residual linear probes and CKA similarity, evaluated on more than 5,000 latent video calibration samples.The video-specific evaluation accounts for temporal motion and spatial appearance interactions, while surrogate transformer blocks replace selected spans.
  • 3.2 Few-Step Generation: Two-stage fine-tuning first restores reliable multi-step motion-effect generation through supervised warm-up, then applies AdvDMD-based adversarial distillation for stable few-step generation.The warm-up uses the standard flow-matching objective and data from both the original dense model and teacher-generated cinematic-effect videos.
  • 3.2 Few-Step Generation: The 4-step generator is refined using DMD, adversarial, and GRPO objectives, with teacher-generated reference videos supplying motion-effect targets.GRPO uses discriminator scores on intermediate latents to refine the 4-step trajectory.
  • Quantization: Hybrid post-training quantization uses layer-wise precision assignment, retaining 16-bit activations and applying FP8 weights to most quantized DiT linear layers.The strategy targets lower runtime memory footprint and latency for on-device generation.

4 Experiments

Experiments show that CineMobile preserves competitive video quality and cinematic controllability with a compact 4-step generator. Ablations and deployment analyses further demonstrate substantial acceleration, reduced memory demands, and improved slow-motion reconstruction.

  • Quantitative analysis: CineMobile matches the teacher and larger models in video quality while using 10% of the parameters and 20% of the denoising steps.Against the 20-step Wan2.1-I2V-14B teacher, absolute total-score gaps are 0.92 on bullet time and 0.66 on dolly zoom.
  • Qualitative analysis: CineMobile produces smoother camera progression and more coherent temporal evolution than several commercial models, especially at later temporal ratios.The comparison identifies reduced temporal fluctuation, more stable subject scale, and more consistent background motion as strengths.
  • Human evaluation: CineMobile remains competitive in perceptual quality and achieves the strongest camera-control result among representative commercial models.Its advantages include artifact control, distortion, uniformity, and temporal consistency, with some competitors slightly stronger on individual dimensions.
  • Efficiency analysis: 40.11× acceleration is achieved across CineMobile’s optimization stages, with step distillation reducing denoising time to 2.45 s and DiT-FLOPs to 4.66×10^14.Structured pruning reduces runtime to 12.28 s, while hybrid precision quantization mainly lowers memory cost and modestly improves runtime.
  • Comparison with frame interpolation: 35.68 dB PSNR and 0.858 SSIM are achieved by CineMobile versus FILM’s 30.57 dB and 0.713 under 20× slow-motion synthesis.Only the first and last frames are provided, and CineMobile shows higher reconstruction quality across the large temporal gap.

5 Conclusion

CineMobile is an efficient on-device image-to-video framework for cinematic motion effects. Experiments show improved inference efficiency, competitive visual quality and temporal consistency, and favorable camera-motion control in human evaluation.

  • Framework: CineMobile provides an efficient on-device image-to-video generation framework for cinematic motion effects.The framework targets bullet time, dolly zoom, and slow motion.
  • Experimental findings: Experiments on bullet time, dolly zoom, and slow motion show substantially improved inference efficiency with competitive visual quality and temporal consistency relative to the teacher model.The comparison covers both generation efficiency and video quality properties.
  • Experimental findings: Human evaluation finds favorable camera-motion control against commercial references.The evaluation specifically concerns cinematic camera-motion effects.

A Effect Definitions

The paper focuses on three cinematic motion effects—bullet time, dolly zoom, and slow motion—each defined by distinct camera-motion or temporal properties. Their generation requires preserving subject appearance, scene structure, and temporal coherence.

  • Effect Definitions: The experiments examine bullet time, dolly zoom, and slow motion as representative cinematic motion effects with different visual requirements.Bullet time and dolly zoom emphasize camera-related motion, whereas slow motion emphasizes temporal continuity and realistic motion.
  • Bullet Time: Bullet time moves the viewpoint rapidly around a relatively frozen subject, requiring coherent camera motion, stable subject identity, and consistent scene structure.Successful videos preserve the foreground subject while producing a smooth orbital or sweeping camera trajectory.
  • Dolly Zoom: Dolly zoom simultaneously changes camera position and focal scale, keeping the subject relatively stable while dramatically changing background perspective.The effect creates a push-pull perception in which the foreground remains anchored as the background expands or contracts.
  • Slow Motion: Slow motion temporally stretches movement so dynamic events unfold more slowly than in real time, emphasizing interpolation, appearance stability, and reduced artifacts.It depends less on explicit camera trajectory control and more on temporal continuity, detail preservation, and motion realism.

B Evaluation Criteria

CineMobile is evaluated with task-aligned automatic dimensions covering consistency, smoothness, and visual quality, while human evaluation assesses artifacts, distortion, uniformity, consistency, and camera control. Motion-amplitude and generic camera-control metrics are excluded when they do not fairly reflect controllable cinematic effects.

  • Automatic evaluation: Automatic evaluation reports seven dimensions spanning subject and background consistency, motion smoothness, aesthetic and imaging quality, and I2V subject and background.Together, these dimensions target identity preservation, background coherence, and temporally smooth, visually plausible motion.
  • Automatic evaluation: Dynamic degree is omitted because motion amplitude does not define bullet time, dolly zoom, or slow motion quality.The metric may favor larger motion rather than coordinated camera transformation, geometric consistency, or temporal smoothness.
  • Automatic evaluation: The VBench camera-control metric is excluded from base-I2V comparisons because Wan2.1-based models are not optimized for the target cinematic motion patterns.Its score may primarily reflect whether models were explicitly adapted to those effects, so selected VBench dimensions provide the automatic comparison.
  • Human evaluation: Human evaluation rates five dimensions: artifact control, distortion, uniformity, consistency, and camera control.These assess rendering artifacts, plausible geometry, frame-to-frame appearance stability, temporal coherence, and adherence to intended motion patterns.

C Compare with Width Pruning

Width pruning causes noticeable identity deviation and detail degradation, motivating CineMobile to preserve the original backbone width and compress through structured depth pruning instead.

  • C Compare with Width Pruning: Width pruning causes identity deviation and detail degradation, so the method preserves backbone width and removes redundant layer transformations through structured depth pruning.This keeps the pruned model aligned with the original backbone’s feature space.

D Additional Qualitative Results

Additional qualitative results cover bullet time, dolly zoom, and slow motion on a MediaTek Dimensity 8400 Ultimate 5G platform. The visualizations show stable subject identity, coherent foreground-background relations, and smooth temporal progression across diverse portrait scenes.

  • The qualitative results cover three cinematic motion effects: bullet time, dolly zoom, and slow motion.
  • All videos in this section are generated on a MediaTek Dimensity 8400 Ultimate 5G platform.
  • Across diverse portrait scenes, CineMobile maintains stable subject identity, coherent foreground-background relations, and smooth temporal progression.Frames are uniformly sampled at different temporal ratios from each generated video.

E Data Curation and Preprocessing

CineMobile’s data curation uses high-quality portrait datasets and an automated filtering pipeline targeting scene suitability and visual quality. Qualitative results show smooth, consistent cinematic motion across bullet time, dolly zoom, and slow motion.

  • Data Curation: Training images come from the high-quality portrait datasets PPR10K and Pixel, with additional curation for cinematic portrait-centric image-to-video generation.The curation pipeline is designed to improve visual quality and scene suitability.
  • Bullet Time: CineMobile produces smooth bullet-time viewpoint transitions while preserving subject appearance and scene geometry across frames.The figure samples uniformly spaced frames at different temporal ratios.
  • Dolly Zoom: Dolly-zoom results maintain consistent foreground stabilization with coordinated background perspective variation.These properties capture the key visual characteristics of the Hitchcock dolly-zoom effect.
  • Slow Motion: Slow-motion results preserve temporal smoothness and subject consistency while reducing abrupt motion changes across scenes.The resulting sequences are visually plausible across different scenes.
Loading 2607.03803v1…