Source-linked AI summary
Slicing Aided Hyper Inference and Fine-tuning for Small Object Detection
Fatih Cagatay Akyon, Sinan Onur Altinuc, Alptekin Temizel
TL;DR
Small and distant objects are difficult to detect because they occupy few pixels, particularly in high-resolution surveillance imagery. The paper introduces a generic slicing-aided inference and fine-tuning framework that works with existing detectors, improving AP across aerial detection benchmarks while exposing a patch-size trade-off for larger objects.
Problem
Small and distant objects contain few pixels and insufficient detail, challenging conventional detectors on high-resolution surveillance imagery.
Method
The framework slices images into overlapping patches, resizes them to enlarge relative object scale, runs detection on patches, and optionally fine-tunes with augmented patch data.
Results
Up to 6.8% AP increase comes from slicing-aided inference, with slicing-aided fine-tuning adding up to 14.5% AP and 25% overlap adding 2.9% AP.
Takeaways & Limitations
Slicing-aided hyper inference can be integrated into any detector without pretraining, while computation increases linearly and memory requirements remain fixed.
Takeaways & Limitations
As patch sizes decrease, larger objects may not fit within slices, potentially reducing large-object detection performance.
Abstract
from arXiv · showhide
Detection of small objects and objects far away in the scene is a major challenge in surveillance applications. Such objects are represented by small number of pixels in the image and lack sufficient details, making them difficult to detect using conventional detectors. In this work, an open-source framework called Slicing Aided Hyper Inference (SAHI) is proposed that provides a generic slicing aided inference and fine-tuning pipeline for small object detection. The proposed technique is generic in the sense that it can be applied on top of any available object detector without any fine-tuning. Experimental evaluations, using object detection baselines on the Visdrone and xView aerial object detection datasets show that the proposed inference method can increase object detection AP by 6.8%, 5.1% and 5.3% for FCOS, VFNet and TOOD detectors, respectively. Moreover, the detection accuracy can be further increased with a slicing aided fine-tuning, resulting in a cumulative increase of 12.7%, 13.4% and 14.5% AP in the same order. Proposed technique has been integrated with Detectron2, MMDetection and YOLOv5 models and it is publicly available at https://github.com/obss/sahi.git .
1. INTRODUCTION
Small, distant objects occupy few pixels and challenge conventional detectors, especially in high-resolution surveillance imagery. The paper proposes slicing-aided inference and fine-tuning to improve their detection while maintaining higher memory utilization.
- Conventional detectors perform well on low-resolution images with relatively large objects but face challenges on high-resolution imagery.
- Figure 1 compares conventional TOOD inference, slicing-aided hyper inference, and slicing-aided hyper inference after fine-tuning.
- Small objects are difficult to detect because they contain few pixels and insufficient visual detail.
- The proposed solution applies slicing-aided inference and fine-tuning to small object detection on high-resolution images.
2. RELATED WORK
Existing object detectors differ in architecture and accuracy, while specialized small-object methods can be difficult to adapt. The paper instead proposes a generic slicing pipeline applicable to existing detectors, with optional fine-tuning for further gains.
- Single-stage detectors are typically faster, whereas two-stage detectors generally provide higher accuracy.
- Anchor-free detectors remove predefined anchor boxes and directly predict object locations from feature-pyramid points.
- General detection algorithms perform poorly on high-resolution images containing small, dense objects.
- Some specialized optimization methods require training from scratch and are difficult to adapt to existing detectors.
- The proposed slicing-aided pipeline can be used with any existing object detector, without fine-tuning for inference, while fine-tuning can provide additional performance gains.
3. PROPOSED APPROACH
The approach slices images into overlapping patches, enlarges their relative object scale, and runs detection independently on resized patches. Fine-tuning augments training data with such patches, while inference merges patch predictions and can optionally include full-image results.
- Dividing images into overlapping patches increases the relative pixel area of small objects presented to the detector.
- Slicing Aided Fine-tuning: During fine-tuning, extracted patches are resized while preserving aspect ratio and added to the training data alongside original images.
- Slicing Aided Fine-tuning: Smaller fine-tuning patches can reduce large-object detection performance when large objects do not fit within a slice or intersecting areas.
- Slicing Aided Hyper Inference: Slicing-aided inference divides an image into overlapping patches, resizes them, and applies detection independently to each patch.
- Slicing Aided Hyper Inference: Patch predictions are mapped back to the original image and merged with non-maximum suppression; full-image inference may also be included.
4. RESULTS
Experiments evaluate the slicing pipeline on VisDrone and xView, datasets containing small objects, using multiple detectors and defined inference or fine-tuning settings. Results show gains from slicing aided inference and fine-tuning, with overlap affecting object-size performance.
- Experimental setup: Experiments use FCOS, VFNet, and TOOD with MMDetection on VisDrone and xView datasets.VisDrone contains small, dense, partially occluded drone-view objects, while xView contains over 1M instances across 60 classes.
- Experimental setup: Table 1 reports mean average precision on the VisDrone19-Detection test-dev set for SF, SAHI, FI, and PO configurations.SF denotes slicing aided fine-tuning, SAHI slicing aided inference, FI full-image inference, and PO overlapping patches.
- Experimental setup: Table 2 reports mean average precision on the xView validation split using the same SF, SAHI, FI, and PO configuration labels.The xView experiments use randomly selected 75% and 25% training and validation splits.
- Detection results: 6.8%, 5.1%, and 5.3% AP increases are obtained from SAHI for FCOS, VFNet, and TOOD, respectively.These gains are reported for the VisDrone evaluation described in the results passage.
- Detection results: 12.7%, 13.4%, and 14.5% cumulative AP increases are obtained for FCOS, VFNet, and TOOD after slicing aided fine-tuning.The reported detector order is FCOS, VFNet, and TOOD.
- Detection results: Overlap between slices increases small/medium-object and overall AP but slightly decreases large-object AP.The increase is attributed to additional small-object true positives, while the decrease is attributed to false positives matching large ground-truth boxes.
5. CONCLUSION
The conclusion presents slicing aided hyper inference as an inference-pipeline addition that avoids pretraining, with configurable computation and memory trade-offs. It reports AP gains from inference, fine-tuning, and patch overlap while noting increased computation time.
- Conclusion: SAHI integrates directly into object-detection inference pipelines without requiring pretraining.The approach is presented as applicable to the tested FCOS, VFNet, and TOOD detectors on VisDrone and xView.
- Conclusion: Up to 6.8% AP increase is reported from slicing aided hyper inference.The conclusion attributes this result to experiments with FCOS, VFNet, and TOOD on VisDrone and xView.
- Conclusion: 14.5% additional AP increase for small objects is reported after applying slicing aided fine-tuning.The conclusion separately reports a further 2.9% AP increase from 25% overlap between slices.
- Conclusion: The approach increases computation time linearly while keeping memory requirements fixed.Patch sizes can be adjusted to trade computation and memory budgets for the target platform.