Source-linked AI summary
Diverse Instance Generation via Diffusion Models for Enhanced Few-Shot Object Detection in Remote Sensing Images
Yanxing Liu, Jiancheng Pan, Jianwei Yang, Tiancheng Chen, Peiling Zhou, Bingchen Zhang
TL;DR
Few-shot remote-sensing object detection is constrained by limited labeled-instance diversity. DIG-FSOD adapts pretrained diffusion models through hybrid image inversion and semantic contrastive learning, achieving reported gains across datasets and baselines.
Problem
Few-shot remote-sensing object detection has limited labeled samples and insufficient instance diversity, restricting existing FSOD methods.
Method
DIG-FSOD uses hybrid image inversion to condition latent diffusion with global and local features, while semantic contrastive learning improves class consistency.
Results
The method improves FSOD performance across datasets and baseline approaches, including average novel-class mAP gains of 7.23% on DIOR with SAE-FSDet and 1.56% on NWPU VHR-10.
Takeaways & Limitations
HIIM significantly improves FSOD performance, and adding class contrastive learning provides further enhancement.
Abstract
from arXiv · showhide
Few-shot object detection (FSOD) aims to detect novel instances with only a limited number of labeled training samples, presenting a challenge that is particularly prominent in numerous remote sensing applications such as endangered species monitoring and disaster assessment. Existing FSOD methods for remote sensing images (RSIs) have achieved promising progress but remain constrained by the limited diversity of instances. To address this issue, we propose a novel framework that can leverage a diffusion model pretrained on large-scale natural images to synthesize diverse remote sensing instances, thereby improving the performance of few-shot object detectors. Instead of directly synthesizing complete remote sensing images, we first generate instance-level slices via a specialized slice-to-slice module, and then embed these slices into full-scale imagery for enhanced data augmentation. To further adapt diffusion models for remote sensing scenarios, we develop a class-agnostic image inversion module that can invert remote sensing instance slices into semantic space. Additionally, we introduce contrastive loss to semantically align the synthesized images with their corresponding classes. Experimental results show that our method hasachieved an average performance improvement of 4.4% across multiple datasets and various approaches. Ablation experiments indicate that the elaborately designed inversion module can effectively enhance the performance of FSOD methods, and the semantic contrastive loss can further boost the performance.
I. INTRODUCTION
Remote-sensing FSOD is limited by scarce labeled data and insufficient instance diversity. DIG-FSOD addresses this gap by adapting pretrained diffusion models with hybrid inversion and semantic contrastive learning.
- Remote-sensing data acquisition and annotation are challenging, while few labeled samples can cause overfitting and low detection performance.
- Existing remote-sensing FSOD methods improve detection but do not increase the diversity of training instances.
- DIG-FSOD inverts condition RSIs into domain-agnostic semantic spaces so pretrained diffusion models can generate high-quality remote-sensing images.
- HIIM combines detail-rich self-supervised features with semantically rich CLIP features to generate high-quality remote-sensing images.
- A semantic contrastive loss aligns condition-image global features with textual semantic features to preserve class consistency.
II. METHOD
The method uses latent diffusion conditioned on global and local features extracted from augmented remote-sensing images. Training combines diffusion reconstruction, textual semantic alignment, and image reconstruction objectives.
- The framework encodes augmented condition images into global and local features, maps images into VAE latent space, and generates latent codes through conditional diffusion.
- The diffusion model is trained with mean-squared error on noisy latent codes conditioned by global and local features.
- A contrastive loss maintains consistency between global image features and textual semantic features.
- Generated images receive reconstruction supervision through mean-squared error and an additional contrastive learning loss.
B. Hybrid Image Inversion Module
HIIM separates semantic and detailed conditioning: a global encoder captures target semantics, while local features provide fine-grained information through cross-attention.
- HIIM contains a global concept encoder for textual latent inversion and a local feature encoder for retaining condition-image details.
- Global and local features are integrated into diffusion through additional cross-attention layers.
- The global encoder combines CLIP features with Q-Former-derived semantic information to better understand the condition image.
- The CLIP [CLS] feature and aggregated semantic features are concatenated to form the global feature f_g.
2) Local feature encoder:
The local feature encoder supplies discriminative, multiscale details that complement global semantics during generation. These features condition the reverse diffusion process through additional cross-attention layers.
- Global features alone may miss discriminative local details, so the method uses DINOv2 multi-level patch features as fine-grained references.
- The framework combines global and local latent features before conditional diffusion generates new latent features and AnyDoor embeds generated instances into large remote-sensing images.
- The local feature extractor produces multiscale detailed image features from the condition image, with five target scales used experimentally.
- Local features condition the reverse diffusion process through additional cross-attention layers.
C. Class Contrastive Learning
The method uses InfoNCE contrastive learning to align generated-instance global features with class-specific textual semantics, improving semantic consistency.
- C. Class Contrastive Learning: The method projects global features into textual semantic space and aligns them with class representations using InfoNCE loss.The projection uses a single-layer perceptron before alignment.
- C. Class Contrastive Learning: The contrastive loss improves consistency between generated instances and semantic classes.This encourages generated instances to capture target-class semantic information.
III. EXPERIMENTS
Experiments evaluate the method on DIOR and NWPU VHR-10 using few-shot novel-class detection and mean Average Precision.
- 1) Datasets:: Experiments use the DIOR and NWPU VHR-10 remote-sensing datasets.DIOR contains 20 categories, while NWPU VHR-10 contains 800 optical images.
- 1) Datasets:: DIOR contains 23,463 images and 192,472 objects across 20 common categories.The dataset is divided into training, validation, and testing sets.
- 1) Datasets:: NWPU VHR-10 contains 800 optical images, including 150 negative and 650 positive samples.Few-shot novel classes include airplane, baseball-diamond, and tennis court.
- 1) Datasets:: Evaluation calculates mean Average Precision (mAP) for few-shot novel classes on each dataset.The evaluation follows SAE-FSDet.
2) Implementation Details:
Implementation uses SAE-FSDet as the primary baseline, Stable Diffusion v1.5 for generation, and a two-phase training process for class-agnostic inversion.
- 2) Implementation Details:: Most experiments use SAE-FSDet, while the data-generation method remains compatible with other FSOD detectors.The implementation uses Stable Diffusion v1.5 pretrained on LAION-5B and CLIP-ViT-G as the text encoder.
- 2) Implementation Details:: Training is divided into two phases, beginning with pretraining a class-agnostic HIIM on combined remote-sensing datasets.The combined datasets include DIOR, DOTA, and FAIR1M.
- 2) Implementation Details:: Effectiveness is evaluated with different FSOD methods on DIOR and NWPU VHR-10.This evaluation tests the method across remote-sensing datasets and detector settings.
1) Results on DIOR Dataset:
The method improves few-shot detection across multiple detectors and datasets, with stronger gains under limited data and increased instance diversity.
- 1) Results on DIOR Dataset:: The method enhances various FSOD approaches, especially when training data is extremely limited.The comparison includes FSCE, MSOCL, and SAE-FSDet across 3-, 5-, 10-, and 20-shot settings.
- 1) Results on DIOR Dataset:: 7.23% average mAP improvement and 12.81% maximum improvement are achieved with SAE-FSDet across four DIOR few-shot settings.The settings are 3-, 5-, 10-, and 20-shot.
- 1) Results on DIOR Dataset:: Generated instances increase few-shot target diversity while preserving discriminative features.Figure 2 compares reference images with corresponding generated instances.
- 2) Results on NWPU VHR-10 Dataset:: The method improves performance across various baseline models on NWPU VHR-10.The comparison includes approaches for natural scenes and remote-sensing images.
- 2) Results on NWPU VHR-10 Dataset:: 1.56% average mAP improvement and 2.36% maximum improvement are achieved with SAE-FSDet on NWPU VHR-10.The improvement diminishes as the number of samples increases because the model approaches its capacity limit.
C. Ablation Studies
Ablation studies on DIOR evaluate how generated-instance augmentation and its components affect few-shot object detection. The results show that HIIM improves FSOD performance, while adding class contrastive loss provides further enhancement.
- The ablation compares SAE-FSDet, text-inversion augmentation, HIIM-based augmentation, and HIIM with additional class contrastive loss under 3-, 5-, 10-, and 20-shot settings.The comparison uses DIOR and reports results in Table III.
- Instances generated using few-shot fine-tuned diffusion improve FSOD performance, but relying solely on CLIP features is insufficient for high-quality instance generation.
- HIIM significantly improves FSOD performance when used for target-instance generation and data augmentation.
- Adding class contrastive loss to HIIM further enhances the performance of the augmented detector.
2) Ablation on HIIM:
The HIIM ablation evaluates feature-extractor choices for global semantics and local details. Replacing the local encoder with DINOv2 yields the strongest reported improvement over the baseline.
- The HIIM ablation compares a baseline, data augmentation, an added local feature encoder, and a DINOv2-based local encoder.The experiments are conducted on DIOR to assess global and local feature-extractor selection.
- Data augmentation effectively improves detection performance on datasets with limited samples.
- Adding a local feature encoder enhances detailed information in the generated images.
- 7.23% average improvement is achieved over the baseline when the local feature encoder is replaced with the more fine-grained DINOv2.