PaperlayerView exact revision on arXiv

Source-linked AI summary

SSMB: Self-Supervised Local Feature Detection under Motion Blur

Zhenjun Zhao, Fabio Bellavia, Wenting Wang, Fan Zhu, Jiajun Wu, Suryansh Kumar, Mingqiang Wei, Haoang Li, Javier Civera

arXiv:2608.27181v1cs.CV

TL;DR

Motion blur harms keypoint repeatability, while deblurring pipelines and handcrafted-detector supervision limit existing approaches. SSMB introduces deblur-free self-supervised detection with local feature enhancement and two-stage training, and it consistently outperforms sparse baselines across motion-blur tasks.

  • Problem

    Motion blur distorts local image structure and degrades keypoint repeatability; existing solutions rely on costly restoration or supervision from handcrafted sharp-image keypoints.

  • Method

    SSMB combines an LDE module with geometric pretraining on synthetic shapes and blur-aware training on real sharp-blur pairs using a multi-component self-supervised objective.

  • Results

    SSMB establishes state-of-the-art performance among sparse detectors and consistently outperforms supervised and self-supervised baselines across four motion-blur tasks.

  • Takeaways & Limitations

    Self-supervised geometric and cross-domain consistency constraints can support deblur-free sparse keypoint detection under motion blur without handcrafted or external pseudo-labels.

Abstract

from arXiv · show

Keypoint detection under motion blur remains a significant challenge, as blur distorts local image structure and degrades the repeatability of feature localization. Existing approaches either rely on computationally expensive deblur-then-detect pipelines that may introduce restoration artifacts, or learn to regress the image positions of handcrafted keypoints extracted on sharp images, which reflects the assumptions of the handcrafted detector rather than what is truly repeatable under blur. We present SSMB, a deblur-free, self-supervised keypoint detector for motion-blurred images that requires neither handcrafted detectors nor external pseudo-labels. SSMB introduces the Local Discriminability Enhancement (LDE) module, which restores fine-grained local discriminability after global feature mixing. Training is performed in two stages. First, geometric pretraining on synthetic shapes bootstraps spatially discriminative keypoint detection without any external detector, just from the rendered geometry. Second, blur-aware training on real sharp-blur image pairs learns blur-invariant detection through a multi-component self-supervised objective that enforces cross-domain consistency, geometric alignment, and spatial coverage. Extensive evaluations on keypoint detection, image matching, relative pose estimation, and visual localization under motion blur demonstrate that SSMB establishes a new state-of-the-art among sparse keypoint detectors, consistently outperforming both supervised and self-supervised baselines across all tasks. Code, models, and datasets will be publicly available upon paper acceptance.

I. INTRODUCTION

Motion blur disrupts local image structure and keypoint repeatability, while existing restoration- or detector-dependent solutions have important limitations. SSMB addresses this with deblur-free self-supervised learning, achieving strong performance across motion-blur tasks.

  • Motion blur smears local image structure, severely degrading keypoint repeatability and the correspondences needed by downstream vision tasks.
  • Deblur-then-detect pipelines are computationally demanding and can propagate restoration artifacts into extracted features, especially under severe blur.
  • SSMB is a deblur-free keypoint detector that requires neither handcrafted detectors nor external pseudo-labels.
  • The Local Discriminability Enhancement module preserves and reinforces fine-grained local cues after global feature mixing to support self-supervised learning under blur.
  • Geometric pretraining on synthetic shapes bootstraps spatially discriminative detection before blur-aware training on real sharp-blur pairs.
  • The multi-component objective combines homographic adaptation, blur consistency, position consistency, and spatial diversity, with spatial diversity preventing collapse to a few locations.
  • SSMB consistently outperforms supervised and self-supervised sparse baselines across keypoint detection, matching, relative pose estimation, and visual localization under motion blur.

II. RELATED WORK

Prior work includes handcrafted and learned sparse detectors, dense matching, and deblurring pipelines. SSMB instead performs deblur-free self-supervised detection without handcrafted supervision, using local enhancement and two-stage training.

  • Handcrafted detectors use manually designed image statistics, with Harris targeting corners, Hessian targeting blobs, and SIFT offering scale and rotation robustness.
  • Learning-based detectors learn repeatable keypoints from data; SuperPoint bootstraps labels from synthetic geometry and homographic adaptation without manual annotation.
  • Later sparse methods explore rotation equivariance, learned stability scoring, deformable descriptors, detector-description decoupling, and real-time efficiency.
  • Dense or semi-dense matchers such as LoFTR establish correspondences without explicit keypoint detection and can operate in weakly textured regions.
  • Deblurring methods restore sharp images before detection but require substantial computation and may introduce artifacts under severe blur.
  • BALF performs real-time deblur-free detection but regresses SIFT keypoints from sharp images, tying its learned representation to handcrafted supervision.
  • SSMB removes external supervision, preserves local cues with LDE, and trains through geometric pretraining followed by blur-aware learning on sharp-blur pairs.

B. Network Architecture

SSMB combines an MLP-based encoder with LDE modules and a detector head that predicts dense keypoint probabilities and sub-pixel offsets. The architecture preserves local cues after global mixing, supporting localization under blur.

  • Encoder: The encoder uses four cascaded blocks with multi-axis global and local spatial mixing, progressively reducing resolution while expanding channels.The feature map is reduced from H × W to H/8 × W/8, with channel dimensions increasing from 3 to 32, 64, 128, and 256.
  • Local Discriminability Enhancement: An LDE module in each encoder block combines depthwise local-pattern extraction with adaptive channel-statistic gating and residual connections.The module applies Layer Norm, multiplies local and blur-aware branches, and adds both inner and outer residuals.
  • Local Discriminability Enhancement: Removing LDE causes the predicted probability map to collapse toward near-zero almost everywhere, indicating its importance for localization under blur.The LDE design adds approximately 50K parameters.
  • Detector head: The detector head separately predicts a full-resolution probability map and sub-pixel offsets, localizing keypoints at probability peaks refined by the offsets.The probability branch uses 64 within-cell positions plus a dustbin, while the offset branch predicts the (x, y) position within each cell.

C. Self-Supervised Training

SSMB learns without manual keypoint annotations through geometric pretraining followed by blur-aware training on sharp-blur pairs. Its objective combines geometric consistency, blur consistency, positional alignment, and spatial diversity.

  • Geometric pretraining: Geometric pretraining uses rendered synthetic shapes and directly derived corner labels to initialize meaningful spatial discriminability without external detectors.Without this initialization, score maps collapse, assigning nearly identical scores across spatial locations.
  • Blur-aware training: Blur-aware training uses shared weights for paired sharp and blurred GoPro images, with the sharp branch providing homographic-adaptation targets and consistency supervision.The total objective combines four loss terms.
  • Blur consistency: The blur consistency loss aligns blurred-image keypoint cells with detached per-cell argmax predictions from the paired sharp image.This hard target directly aligns predicted keypoint cells without jointly optimizing the sharp target.
  • Position consistency: The position consistency loss applies ℓ2 regression to make sharp and blurred branches predict the same sub-pixel offsets.The offsets represent positions within each detector cell.
  • Spatial diversity: Spatial diversity partitions the feature map into a grid and penalizes low maximum response in any cell, encouraging at least one confident keypoint per region.Despite its small weight, λdiv = 0.005, this term is identified as the most critical component because removing it causes detection collapse.

D. Implementation Details

Implementation uses synthetic geometric pretraining followed by GoPro blur-aware training, with synchronized geometric augmentation and independent photometric augmentation.

  • Stage 1: Stage 1 pretraining uses 25,000 synthetic geometric images for 10 epochs with 320 × 320 crops, batch size 8, Adam, and initial lr = 10^-5.The learning rate uses linear warmup over four epochs.
  • Stage 2: Stage 2 uses 2,912 GoPro sharp-blur pairs from 30 sequences for 36 epochs, with 100 homographies per image and an 8 × 8 spatial-diversity grid.Training uses 320 × 320 crops, rotations up to ±90°, and learning-rate decay at 60% and 80% of training.
  • Augmentation: Stage 2 applies synchronized geometric augmentation followed by independent color jitter and random grayscale conversion to each sharp-blur pair.All training is conducted on a single NVIDIA GeForce RTX 3090.

IV. EXPERIMENTS

The experiments evaluate motion-blur robustness using blurred variants of HPatches, deblurred comparisons, simulated rapid camera motion, and visual localization under synthetic blur.

  • Keypoint detection and matching: Blur-HPatches derives motion-blurred images from HPatches sequences with viewpoint and illumination changes and provides ground-truth homographies.The benchmark contains 116 sequences, each with one reference and five target images.
  • Deblur comparison: Deblur-HPatches applies SRN-DeblurNet and DeblurGAN-v2 without fine-tuning to compare deblur-then-detect pipelines against deblur-free detection.The dataset tests whether restoration can match the performance of direct blurred-image processing.
  • Relative pose estimation: ArchViz evaluates relative pose under rapid simulated camera motion using paired sharp and blurred images with texture-less regions and repetitive patterns.Its evaluation set contains 3,321 image pairs at 768 × 480 resolution.
  • Visual localization: Aachen Day-Night evaluates visual localization with synthetic motion blur applied to query and database images under the established benchmark protocol.The benchmark includes 824 daytime and 98 nighttime queries against 4,328 daytime reference images.

B. Evaluation Metrics

The evaluation measures keypoint repeatability, image matching, relative pose estimation, and visual localization, comparing SSMB with sparse and detector-free baselines under stated protocols.

  • Metrics: Repeatability is the ratio of mutually corresponding keypoints to the smaller detection count, accepting region matches when ϵIoU < 0.4.Each image uses 1,000 keypoints.
  • Metrics: Image matching uses Mean Matching Accuracy at 3, 5, and 10-pixel thresholds, with up to 2,048 keypoints per image.
  • Metrics: Relative pose estimation reports pose-error AUC at 5°, 10°, 20°, and 30° thresholds using five-point essential-matrix estimation with RANSAC.Pose error is the maximum angular error in rotation and translation.
  • Metrics: Visual localization reports the percentage of queries localized within (0.25m, 2°), (0.5m, 5°), and (1.0m, 10°) error thresholds.
  • Baselines: Comparisons include classical, learning-based, and detector-free methods, with SSMB downstream evaluation using HardNet or HyNet descriptors and MNN matching.Detector-free methods operate on full image pairs and are reported separately because their computational cost and downstream targets differ from sparse detectors.

D. Keypoint Detection

On motion-blurred HPatches, SSMB remains stable across blur severity and outperforms sparse alternatives in keypoint detection and image matching, including comparisons with deblur-then-detect pipelines.

  • Sharp-Image Evaluation: SSMB remains competitive on the original all-sharp HPatches dataset despite being designed for motion-blurred images.Full sharp-image results are provided in the appendix.
  • Keypoint Detection: SSMB achieves 77.24%, 77.18%, and 77.16% repeatability on EASY, HARD, and TOUGH blur-to-sharp settings, respectively.It outperforms BALF at every difficulty level, while BALF drops from 74.12% to 71.84%.
  • Keypoint Detection: SSMB achieves 74.20%, 74.33%, and 74.48% repeatability on EASY, HARD, and TOUGH blur-to-blur settings, respectively, outperforming all methods.Its performance slightly improves as blur severity increases.
  • Deblur Comparison: Under TOUGH deblur-to-sharp evaluation, DeblurGAN-v2 with SuperPoint reaches 58.22%, while SSMB directly processes blurred images.The comparison evaluates whether deblur-then-detect can match deblur-free detection.
  • Image Matching: SSMB with HardNet achieves 40.10%, 51.43%, and 77.22% MMA at 3, 5, and 10 pixels overall, ranking first among sparse methods.At 3 pixels under illumination changes, SSMB+HardNet reaches 62.02% versus 34.58% for BALF+HardNet.
  • Image Matching: SSMB also outperforms all detector-free methods in overall MMA at the strict 3-pixel threshold, despite detector-free methods operating on complete image pairs.

F. Relative Pose Estimation

SSMB transfers blur-robust detection to relative pose estimation and visual localization while its ablations identify spatial diversity, blur consistency, geometric pretraining, and LDE as important design choices.

  • Relative Pose Estimation: SSMB achieves top-two results among sparse detectors across both ArchViz blur settings and all reported pose-error thresholds.SSMB+HyNet reaches an AUC of 66.61 at 30° in blur-to-sharp evaluation, highest among sparse methods.
  • Visual Localization: SSMB outperforms all sparse baselines on Aachen Day-Night with motion blur in both daytime and nighttime localization conditions.
  • Computational Comparison: Detector-free matching can require substantially higher computation because it operates on complete image pairs, unlike SSMB’s sparse detector.RoMa v2 could not be evaluated on Aachen because dense correspondence estimation exhausted available system memory during bundle adjustment.
  • Ablation Study: Removing Ldiv drops TOUGH repeatability from 77.16% to 64.07% blur-to-sharp and from 74.48% to 65.26% blur-to-blur, making spatial diversity the most critical loss component.Without Ldiv, detections collapse onto a handful of salient locations.
  • Ablation Study: Skipping synthetic pretraining reduces repeatability from 77.16% to 59.58% blur-to-sharp and from 74.48% to 61.44% blur-to-blur.The authors report that geometric pretraining is an essential prerequisite for escaping score-map collapse during blur-aware training.
  • Ablation Study: Removing LDE reduces repeatability from 77.16% to 33.83% blur-to-sharp and from 74.48% to 50.13% blur-to-blur, despite saving only 0.05M parameters.The authors attribute this to global feature mixing destroying local gradient structure needed to identify discriminative positions.

J. Qualitative Results

SSMB produces well-localized, consistently distributed keypoints on real-world blurred images and repeatable matches across sharp-blur pairs despite viewpoint changes and appearance gaps.

  • SSMB produces well-localized keypoints concentrated on genuine image structure in real-world blurred images, whereas several baselines miss or scatter detections.The comparisons cover RWBI and real camera motion blur.
  • The LDE module preserves dense, fine-grained responses tracing structures such as signage, window frames, and shop dividers.Without LDE, responses collapse toward near-zero across most of the image.
  • SSMB detects well-localized and repeatable keypoints from both sharp and blurred RealBlur images despite cross-domain appearance and viewpoint changes.Qualitative matching uses HardNet descriptors and MNN matching.
  • SSMB achieves state-of-the-art performance among sparse detectors under motion blur across keypoint detection, image matching, relative pose estimation, and visual localization.

APPENDIX

The appendix expands evaluation across sharp, blurred, deblurred, and real-world data, showing strong repeatability and matching performance across settings and thresholds.

  • Additional analyses: The appendix adds complete sharp-HPatches repeatability, disaggregated Blur-HPatches results, full-threshold matching, pose examples, and RealBlur qualitative results.
  • Evaluation tables: The appendix reports repeatability tables for sharp HPatches, Blur-HPatches, and deblurred images under deblur-to-sharp and deblur-to-deblur settings.
  • HPatches: 75.17% overall repeatability on sharp HPatches ranks SSMB second and exceeds BALF at 70.28%.SSMB was designed for blurred images and trained on GoPro blur-sharp pairs.
  • Blur-HPatches: Under Blur-HPatches blur-to-blur evaluation, SSMB’s viewpoint repeatability remains around 64–67% while illumination repeatability exceeds 84%.Illumination repeatability peaks at 84.55% on TOUGH.
  • Blur-HPatches: SSMB outperforms BALF by 3.72 on EASY, 5.90 on HARD, and 6.77 on TOUGH in blur-to-blur evaluation.The margin widens as blur severity increases.
  • Image matching: SSMB with HardNet consistently outperforms sparse methods on MMA curves and surpasses RoMa v2 and MatchFormer at the strict 3-pixel threshold.The comparison includes sparse detector-plus-matcher and detector-free methods.

C. Relative Pose Estimation

Additional qualitative results examine relative-pose matching and detection on real-world blurred images, where SSMB maintains localized keypoints and geometrically consistent correspondences.

  • Relative pose estimation: ArchViz comparisons mark correct matches in green and epipolar errors beyond 5 × 10^-4 in red under blur-to-blur viewpoint changes.
  • Detection: On RealBlur, SSMB maintains dense, well-aligned detections in regions that most baselines fail to cover.The comparison includes eleven methods, including Key.Net, REKD, DeDoDe v2, and XFeat.
  • Matching: Across four indoor and outdoor RealBlur scenes, SSMB yields dense and geometrically consistent matches despite real camera motion blur and viewpoint changes.SSMB is combined with HardNet and MNN matching.
Loading 2608.27181v1…