PaperlayerView exact revision on arXiv

Source-linked AI summary

UAV Video Deblurring via Motion-Aware Diffusion: A Path to Robust Target Detection

Zhiqiang Hu, Shouren Huang, Masatoshi Ishikawa

arXiv:2608.15259v1cs.CVcs.AIcs.RO

TL;DR

UAV video motion blur challenges accurate target detection, while existing methods struggle with dynamic motion and computational demands. The paper introduces motion-adaptive latent scaling with multi-frame alignment and learnable gating, achieving 35.12% AP and 64.70% AP50 while improving deblurring and detection.

  • Problem

    Severe, variable motion blur in UAV videos degrades target detection, while traditional deblurring methods struggle with rapid motion and dynamic environments.

  • Method

    The framework adaptively selects latent resolution from optical-flow motion intensity and aligns and gates preceding frames to preserve relevant temporal information.

  • Results

    35.12% AP and 64.70% AP50 are achieved, outperforming all baselines while producing sharper, less artifact-prone restorations.

  • Takeaways & Limitations

    The method supports robust UAV object detection under challenging motion conditions while balancing deblurring quality and inference efficiency.

Abstract

from arXiv · show

Unmanned Aerial Vehicles (UAVs) play a crucial role in various scenarios ranging from disaster response to traffic surveillance. However, aerial video footage often suffers from severe motion blur due to rapid flight maneuvers, vibrations, and camera panning, which can significantly degrade downstream tasks such as target detection. Our goal is to explore a computationally-efficient and effective video deblurring approach to enhance UAV target detection performance. To reduce computational cost, we first propose an Adaptive Latent Scale Selector that dynamically adjusts the latent space resolution according to the intensity of UAV motion, thus balancing detail preservation with inference efficiency. To ensure temporal consistency, we introduce a Multi-Frame Alignment and Learnable Gating module to warp and gate the preceding frames, allowing the model to fuse only relevant temporal information and suppress misaligned or uninformative features. Our method can effectively recover sharp details from the UAV video stream. Extensive experiments on real UAV benchmarks demonstrate that our method not only yields superior deblurring performance but also significantly boosts target detection accuracy, making it highly applicable to robust aerial vision tasks.

I. INTRODUCTION · II. RELATED WORKS · A. UAV Object Detection

UAV videos suffer motion blur that weakens target detection, while existing deblurring and detection methods face limitations in dynamic, resource-constrained scenarios. The proposed framework adapts latent resolution to motion and aligns and gates temporal information to improve clarity and detection.

  • I. INTRODUCTION: UAVs support disaster surveillance, urban monitoring, and environmental inspection, where detecting pedestrians, vehicles, and infrastructure anomalies enables timely decisions.UAV-captured videos can suffer severe motion blur from rapid maneuvers, vibrations, and dynamic environmental factors.
  • I. INTRODUCTION: Traditional video deblurring struggles with unpredictable environments, rapid UAV motion, complex temporal modeling, and diverse real-world blur types.
  • II. RELATED WORKS: EDVR uses deformable convolution and gradual refinement for temporal alignment and motion compensation but primarily focuses on spatial information.This limitation can reduce effectiveness under complex, unpredictable motion blur in UAV detection and tracking.
  • II. RELATED WORKS: Diffusion Models generate high-fidelity images through iterative denoising from Gaussian noise and avoid common GAN issues such as optimization instability or mode collapse.
  • I. INTRODUCTION: The Adaptive Latent Scale Selector dynamically adjusts latent feature scale using optical-flow motion intensity, preserving details during rapid motion while reducing computation elsewhere.Smaller latent features are selected for smooth motion and larger features for intense motion, producing shorter and longer inference times, respectively.
  • I. INTRODUCTION: The framework aligns preceding frames and uses learnable gating to filter misaligned or degraded cues, recovering temporal details while mitigating ghosting and jitter.The stated objective is to enhance video clarity and boost target detection performance.
  • A. UAV Object Detection: ViT-YOLO captures global context but has substantial computational cost, whereas Drone-YOLO optimizes feature extraction for UAV detection.These approaches address detection efficiency, while the proposed work explicitly reduces blur to enhance detection.

B. Video Deblurring … A. Method Overview

The section reviews spatial-temporal and transformer-based video deblurring, then motivates diffusion models while identifying their computational and temporal-consistency challenges. It formulates UAV video deblurring as producing a sequence that preserves spatial details and temporal consistency for downstream applications.

  • B. Video Deblurring: Deep video deblurring methods exploit spatial and temporal coherence, extending beyond single-image techniques through cross-frame alignment and aggregation.
  • B. Video Deblurring: Transformer-based approaches model long-range dependencies and use recurrent guided deformable attention or flow-guided sparse attention to leverage spatial-temporal information.
  • C. Diffusion Models for Low-Level Vision: Diffusion models support image restoration tasks such as super-resolution, inpainting, and deblurring through iterative denoising that generates high-fidelity images.
  • A. Method Overview: The framework encodes each blurry frame at multiple scales and uses optical-flow motion intensity to select an appropriate latent feature size.
  • C. Diffusion Models for Low-Level Vision: Applying diffusion models to high-resolution and video restoration is hindered by significant computational cost and difficulty maintaining temporal consistency across frames.
  • A. Method Overview: The method targets a blurry UAV video sequence Iblur = {Iblur 1, Iblur 2, . . . , Iblur N}, with each frame represented in RH×W ×C.
  • A. Method Overview: Its goal is to produce a deblurred sequence Iclear = {Iclear 1, Iclear 2, . . . , Iclear N} that preserves spatial details and temporal consistency for downstream UAV applications.

B. Adaptive Latent Scale Selector · 1) Latent Extraction: · 2) Adaptive Latent Scale Selector:

The method extracts multi-level latent features and adaptively selects each frame’s latent resolution from optical-flow motion intensity. Smoothed flow-based scaling preserves detail during strong motion while avoiding unnecessary high-resolution encoding during smooth flight.

  • 1) Latent Extraction:: Each frame I_i is encoded at a chosen latent scale s into a latent representation z_i^(s,t).The scale belongs to {1, . . . , S}, while i indexes frames and t denotes the diffusion step.
  • 1) Latent Extraction:: The encoder E_s produces latent representations at resolution scale s using Stable Diffusion’s auto-encoder architecture.The architecture provides multi-level latent features.
  • 2) Adaptive Latent Scale Selector:: For varying UAV motion, the selector chooses each frame’s latent resolution scale from optical flow between consecutive blurry frames.RAFT computes forward flow f_i−1→i from frame i−1 to frame i.
  • 2) Adaptive Latent Scale Selector:: Motion intensity is defined as the magnitude ∥Flow(I_i^blur)∥ of the optical-flow vector.The flow vector comprises horizontal and vertical components (Δx, Δy).
  • 2) Adaptive Latent Scale Selector:: The selector linearly interpolates motion intensity within predefined scales {1, . . . , S}, then rounds to obtain the selected scale s_i.This replaces reliance on a fixed threshold.
  • 2) Adaptive Latent Scale Selector:: Larger flow magnitude indicates stronger motion and selects a larger latent scale s.The mapping dynamically adjusts resolution according to motion intensity.
  • 2) Adaptive Latent Scale Selector:: The adaptive reference blends current and previous flow magnitudes with smoothing factor α=0.8 in the experiments.The selected scale adapts in real time, preserving fine details during large movement and avoiding unnecessary high-resolution encodings during smooth flight.

C. Latent Diffusion with Motion-Aware Conditioning · 1) Multi-Frame Alignment with Learnable Gating:

The method aggregates preceding UAV video frames while addressing motion-induced misalignment and occlusion. It resizes latent features, warps them using optical flow, gates them with learnable masks, and stacks the results for cross-attention.

  • 1) Multi-Frame Alignment with Learnable Gating:: High-dynamic UAV conditions cause severe motion variations, environmental complexity, frame misalignment, and occlusion.These discrepancies make naive multi-frame aggregation prone to artifacts and temporal inconsistencies.
  • 1) Multi-Frame Alignment with Learnable Gating:: The aggregation module preserves consistency in the current blurred frame while exploiting clear-region information from preceding frames.It is designed to address the limitations of direct multi-frame aggregation.
  • 1) Multi-Frame Alignment with Learnable Gating:: Latents at scales different from the target scale are first resized before temporal alignment.A latent at scale s′ ≠ s is resized to scale s.
  • 1) Multi-Frame Alignment with Learnable Gating:: Optical flow f(i−m)→i is used to warp each preceding latent toward the current frame.The warping operation aligns latent features from preceding frames with frame i.
  • 1) Multi-Frame Alignment with Learnable Gating:: A three-layer convolutional network with 7×7, 5×5, and 3×3 kernels serves as the warping operator W.The operator uses progressively smaller convolutional kernels for warping.
  • 1) Multi-Frame Alignment with Learnable Gating:: A learnable mask Γi_m ∈ R^H×W gates the warped latent features.The mask is applied to obtain the gated latent.
  • 1) Multi-Frame Alignment with Learnable Gating:: A lightweight gating network predicts Γi_m using a sigmoid function and elementwise multiplication.The sigmoid σ(·) produces the gating mask, while ⊙ denotes elementwise multiplication.
  • 1) Multi-Frame Alignment with Learnable Gating:: The gated latents are stacked for the subsequent Multi-Frame Cross-Attention calculation.Stacking occurs after alignment and learnable gating.

2) Latent Fusion with Multi-Frame Cross-Attention (MFCA):

The method fuses information from past frames through Multi-Frame Cross-Attention (MFCA), projecting flattened inputs into a common space and reshaping the output into a refined 2D latent grid.

  • Latent Fusion with Multi-Frame Cross-Attention (MFCA): Multi-Frame Cross-Attention (MFCA) fuses information from past frames.The operation is illustrated in Figure 3(c).
  • Latent Fusion with Multi-Frame Cross-Attention (MFCA): W_Q, W_K, and W_V map each flattened input into a common d-dimensional space.These mappings define the shared representation used by MFCA.
  • Latent Fusion with Multi-Frame Cross-Attention (MFCA): The MFCA output is reshaped into a 2D grid to obtain the refined latent z_i.The reshaping restores the spatial grid structure of the latent representation.

3) Scale-Conditioned Reverse Denoising:

The method conditions reverse denoising on both spatial scale and a refined latent prior. A UNet-like denoiser then performs standard reverse diffusion to produce and decode the deblurred frame.

  • Scale conditioning: A learned scale embedding converts the discrete scale index into a feature vector injected into UNet conditional normalization layers.This enables explicit handling of different spatial scales during denoising.
  • Latent-prior conditioning: The refined latent is transformed into adaptive conditional-normalization parameters, providing structural information from the original input as an additional deblurring prior.The refined latent modulates features alongside the scale-conditioned representation.
  • Denoiser inputs: The UNet-like denoiser receives the refined latent, diffusion timestep, scale embedding, and optionally the clean latent.Scale information and the refined latent are incorporated through conditional normalization.
  • Reverse diffusion and decoding: Standard reverse diffusion updates are iterated until t = 0 to obtain the deblurred latent, which is decoded into the deblurred frame.The final latent is passed through decoder D to yield Iclear.

D. Training Strategy

The training strategy proceeds in three phases: ground-truth auto-encoder pre-training, multi-scale diffusion denoising training with temporal cues, and joint fine-tuning of all modules under a combined objective.

  • Phase 1: Pre-training Encoder and Decoder: Phase 1 pre-trains an auto-encoder on ground-truth data, producing a VAE that encodes images into latent z(s,0) and decodes them back.The reconstructed output is trained to closely match the ground-truth frame.
  • Phase 2: Multi-Scale Diffusion Denoising Model Training: Phase 2 randomly samples latent scale s from {1, . . . , S}, applies T-step forward diffusion, and trains denoising with fused multi-frame latents.The model injects scale embeddings and refined latents, optionally combined with clean latents, to handle diverse resolutions and temporal cues.
  • Phase 3: Joint Fine-Tuning: Phase 3 jointly fine-tunes the scale-specific encoders, decoder, and diffusion UNet using a combined reconstruction and denoising objective.The objective balances Lrec and Ldenoise with weights λ1 and λ2.

IV. EXPERIMENTS … C. Main Results

Experiments evaluate deblurring on GoPro, DVD, and VisDrone-VID2019, with UAV footage synthetically blurred for restoration and detection assessment. Training uses a fixed RAFT flow estimator, and results show improved temporal consistency and object detection with the proposed method.

  • A. Datasets: GoPro provides 3,214 blurry-sharp image pairs at 1280 × 720, split into 2,103 training and 1,111 testing pairs.It captures real-world dynamic scenes with significant motion blur.
  • C. Main Results: Ours produces smoother temporal profiles than VRT, RVRT, and Shift-Net+, reducing the distortions, discontinuities, and flickering artifacts observed across consecutive DVD frames.The comparison evaluates temporal consistency along a fixed row.
  • A. Datasets: VisDrone-VID2019 contributes 79 real-world UAV video sequences totaling 33,366 frames from diverse weather, illumination, urban locations, and scene complexities.The benchmark covers a broad range of motion patterns relevant to aerial footage.
  • A. Datasets: Artificial blur is generated from original VisDrone-VID2019 videos to evaluate restoration of high-speed UAV footage and subsequent object detection using MS COCO-style metrics.Reported metrics include AP, AP50, AP75, AR1, AR10, AR100, and AR500.
  • B. Implementation Details: Training uses PyTorch on 8 NVIDIA A100 GPUs with batch size 8, learning rate 4 × 10−4, Adam optimization, and 256 × 256 augmented crops.The RAFT-based flow estimator uses pretrained weights and remains fixed during training.
  • C. Main Results: Ours + RTDETR achieves the best VisDrone-VID2019 detection performance, recovering motion-degraded objects while preserving fine details and outperforming prior methods in precision-recall curves.The comparison includes blurred inputs, RVRT-deblurred frames, and original clear frames, with STFAN, VRT, RVRT, and Shift-Net among the prior methods.

1) Quantitative Evaluation: … E. Results and Discussion

The method delivers strong deblurring performance on GoPro and DVD, with sharper, less artifact-prone restorations enabled by temporal alignment and gating. On VisDrone-VID2019, deblurring substantially improves RTDETR detection, and the proposed method achieves the best reported detection results.

  • 1) Quantitative Evaluation:: Over 0.2 dB relative PSNR gain is obtained against ShiftNet+ on GoPro.Visual comparisons further show sharper textures and reduced artifact levels.
  • 1) Quantitative Evaluation:: 35.18 dB PSNR and 0.9778 SSIM are achieved on DVD, maintaining a clear advantage over prior works.
  • 2) Qualitative Results:: Sharper boundaries, fewer artifacts, and more perceptually faithful textures are observed on GoPro and DVD restorations.MALG integrates spatial and temporal cues to suppress ghosting and jitter in dynamic scenes with complex motion.
  • D. Experimental Result on UAV Object Detection Dataset: RTDETR on blurry VisDrone-VID2019 videos achieves only 21.53% AP, whereas deblurring pipelines consistently improve AP and AR under identical inference settings.The comparison uses Method A for blurry input and Methods B to F for deblurred input.
  • E. Results and Discussion: 33.80% AP is achieved by Shift-Net+ (E), exceeding the 29.22% AP obtained by STFAN (B).VRT (C) and RVRT (D) further refine temporal alignment and improve results.
  • E. Results and Discussion: 35.12% AP and 64.70% AP50 are achieved by the proposed method (F), outperforming all baselines.The improvement is attributed to robust multi-frame alignment and learnable gating that filters misaligned features.

F. Ablation Study … V. SUMMARY

The ablation studies show that adaptive latent scaling, cached temporal context, alignment, and gating jointly improve deblurring and detection while balancing quality against inference cost. Across UAV benchmarks, the motion-aware diffusion framework consistently enhances detection accuracy and deblurring robustness for dynamic real-time applications.

  • F. Ablation Study: The ablation study evaluates the Adaptive Latent Scale Selector and cached latent-frame count to measure their individual contributions to overall model performance.Results are summarized in Table IV.
  • 1) Impact of Adaptive Latent Scale Selector (Latent Scale Ablation):: 34.20 dB PSNR and 32.80% AP are achieved in 35 ms with ALSS, approaching 64 × 64 fixed-scale quality while reducing inference time.Larger fixed latent sizes generally improve PSNR and AP but increase inference time.
  • 1) Impact of Adaptive Latent Scale Selector (Latent Scale Ablation):: ALSS dynamically balances detail preservation and speed by adapting latent resolution to variable UAV motion intensities.This makes it an effective trade-off for UAV scenarios with highly variable motion.
  • 2) Impact of Cached Latent Frames (Multi-Frame Ablation):: 35.12% AP is reached at M = 8 cached past latents, although inference time increases to 40 ms.Increasing M from 1 to 8 improves PSNR, SSIM, and detection AP; gains diminish beyond M = 8, with a slight AP drop at M = 16.
  • 2) Impact of Cached Latent Frames (Multi-Frame Ablation):: Increasing cached latent frames enriches temporal context, improving deblurring and object detection until the benefits diminish beyond M = 8.M = 8 offers the most favorable balance between performance and inference cost.
  • 3) Ablation on Each Module:: ALSS, MAlign, and learnable gating respectively improve motion-adaptive restoration, temporal artifact reduction, and suppression of occluded or misaligned regions.The modules are ablated on both GoPro and VisDrone-VID2019 datasets.
  • V. SUMMARY: The combined model attains the best results across GoPro and VisDrone-VID2019, while experiments on DVD further demonstrate robustness against state-of-the-art deblurring techniques.VisDrone-VID 2019 evaluations confirm improved detection accuracy, supporting real-time use in dynamic environments.
Loading 2608.15259v1…