PaperlayerView exact revision on arXiv

Source-linked AI summary

Example-based Robust Abnormality Detection with Minimal Annotations using Exemplar Med-DETR

Sheethal Bhat, Bogdan Georgescu, Awais Mansoor, Mathias Zinnen, Pranjal Sahu, Florin C. Ghesu, Sasa Grbic, Andreas Maier

arXiv:2608.24281v1cs.CV

TL;DR

Robust medical object detection still requires costly annotations, while medical imaging lacks the grounded text needed by many vision-language methods. The paper extends EM-DETR with exemplar-based feature generation and domain-aware contrastive optimization, achieving near-SOTA CXR detection with fewer than 10% of annotations and limited-retraining support for new findings.

  • Problem

    Medical object detectors need costly expert annotations, and scarce grounded text limits vision-language few-shot detection in the medical domain.

  • Method

    The paper extends EM-DETR with exemplar-based prototype generation and domain-aware iterative negative sampling for few-shot CXR abnormality detection.

  • Results

    Near-SOTA detection performance is achieved on CXR images using fewer than 10% of available annotations across proprietary and public datasets.

  • Takeaways & Limitations

    The framework supports scalable minimal-annotation detection and is structured to accommodate new disease classes with limited retraining.

  • Takeaways & Limitations

    EM-DETR is memory-intensive, and its text encoder primarily acts as a pseudo-class identifier rather than providing meaningful linguistic grounding.

Abstract

from arXiv · show

Reducing annotation requirements remains a key challenge in developing robust medical object detectors. To address this, Vision-Language (VL) object detection methods leverage grounding text information to enable powerful zero-shot and few-shot object detectors in the natural image domain [1, 2, 3, 4]. However, transferring these methods to the medical domain is challenging due to the absence of comparable quality and quantity of the grounding data. Regardless, significant contextual and non-imaging information exists in medical images that remains underutilized. Few-shot learning (FSL) techniques partially address this limitation but struggle to general ize to unseen medical findings and require extensive retraining when new findings are introduced [5, 6]. To overcome these challenges, we extend our prior EM-DETR framework [7] and introduce a scalable FS detection approach designed for efficient abnormality detection in Chest X-Ray (CXR) images under minimal supervision. The proposed architecture incorporates exemplar-based feature generation and domain-aware contrastive optimization, enabling effective adaptation to novel disease findings without exhaustive retraining. Our method achieves near state-of-the-art (SOTA) detection performance using less than 10% of the annotated data, demonstrating its potential for practical, annotation-efficient clinical deployment across both proprietary and public CXR datasets.

Example-based Robust Abnormality Detection with Minimal Annotations using Exemplar Med-DETR

Exemplar Med-DETR targets robust chest-radiograph abnormality detection with minimal annotations. It emphasizes scalable exemplar-based adaptation, extension to new findings, and evaluation across CXR datasets.

  • Exemplar Med-DETR is presented as an example-based approach for abnormality detection with minimal annotations.
  • The framework is designed to scale exemplar-based detection to large chest-radiography datasets while reducing reliance on extensively labeled data.
  • Its structure could support adding new disease classes with limited retraining and future continuous learning without catastrophic forgetting.
  • Evaluations include proprietary and public CXR datasets, ablation studies, and minimal-annotation and out-of-distribution settings.

1. Introduction

The introduction identifies expert annotation burden and limited grounding data as barriers to annotation-efficient CXR detection. It motivates Exemplar Med-DETR as an adaptable, domain-aware framework for learning new findings from few labels.

  • Expert medical-image annotation is time-consuming and institutionally variable, making reduced annotation effort a central CAD research objective.
  • Existing weakly supervised, semi-supervised, active-learning, and few-shot methods have not demonstrated competitive spatial precision at scale in medical imaging.
  • Vision-language detection transfers poorly to medicine because radiology reports rarely contain explicit fine-grained grounding descriptions and comparable grounded data are scarce.
  • CXR systems commonly depend on many annotations per finding and handcrafted NMS, while long-tailed datasets leave many findings sparsely represented.
  • EM-DETR uses learned visual exemplars rather than grounded radiology captions, reducing dependence on dense annotations and avoiding post-processing.
  • The proposed architecture adds class-specific prototype embeddings to a Grounding-DINO-inspired cross-attention pipeline and introduces domain-specific adaptations for CXR data.
  • The contributions include scaling with few labels, adding disease classes with limited retraining, domain-aware iterative negative sampling, and evaluation across CXR datasets.

2. Related Works

Related work spans annotation-efficient learning, multimodal vision-language systems, and few-shot detection. The paper argues that these approaches do not readily provide robust multi-finding few-shot detection for CXR abnormalities.

  • Medical AI annotation-reduction strategies include weakly or semi-supervised learning, multimodal learning, and few-shot learning across classification, detection, and segmentation.
  • Weakly and semi-supervised methods use unlabeled or weakly labeled data through pseudo-labels generated by clustering, consistency regularization, or contextual label generation.
  • Medical imaging differs from general vision through stronger domain shift, spatial-resolution constraints, and class imbalance, while CXR detection requires localizing subtle findings.
  • Vision-language methods use image-caption pretraining and contrastive image-text learning to enable text-conditioned or open-vocabulary detection in natural images.
  • Few-shot multimodal models achieve strong results in medical VQA and related tasks, but CXR few-shot detection remains constrained by scarce grounded text and bounding-box data.
  • FS-DETR and Meta-DETR extend DETR for few-shot object detection using visual prompts or inter-class correlations, primarily in natural-image benchmarks.
  • The reviewed approaches often target natural-image classification or segmentation and do not readily translate to robust few-shot detection of multiple CXR findings.

3. Data preparation

The study evaluates exemplar-based detection first on a small anatomy dataset, then on larger CXR datasets containing findings of varying difficulty under minimal-shot conditions.

  • 3.1. Proprietary CXR Anatomy dataset: The anatomy feasibility study uses 100 anonymized DICOM images, split patient-wise into 80 frontal training images and 20 validation images.The images contain segmentation masks for seven major organs.
  • 3.1. Proprietary CXR Anatomy dataset: Seven organ segmentation masks are converted to bounding boxes, split into left and right classes, and further divided for ribs and spine, producing 10 classes.Ribs and spine are divided into upper, middle, and lower regions using thirds of the original bounding-box height.
  • 3.1. Proprietary CXR Anatomy dataset: Five training images contain heart annotations to simulate the few-shot anatomy setting.
  • 3.2. Proprietary CXR Abnormality dataset: Dataset B contains 18,681 anonymized frontal CXR images with expert bounding-box annotations for 16 findings and validation sets for pleural effusion, pneumothorax, and lesion detection.The validation sets contain 331, 412, and 412 images, respectively.
  • 3.2. Proprietary CXR Abnormality dataset: The abnormality experiments exclude three target findings from base pretraining, then use smaller positive subsets and sufficient negatives for finding-specific finetuning.For pneumothorax, an example finetuning set uses 200 positive and 800 negative images, representing 7% of the entire training dataset.
  • 3.2. Proprietary CXR Abnormality dataset: A public VinDR-CXR test set is evaluated without further finetuning for pneumothorax classification and FROC performance.The test set contains 3,000 images, 22 local labels, 6 global labels, and only 18 pneumothorax-positive samples.

4. Method

EM-DETR adapts exemplar-based detection for minimal-shot CXR abnormality detection through exemplar generation and iterative contrastive training. Class prototypes are refined from ROI features, combined with text embeddings, and used across modular finetuning and inference stages.

  • EM-DETR combines exemplar-based finetuning with iterative contrastive training for minimal-shot detection.The framework adapts the earlier EM-DETR model and uses per-finding exemplars plus iterative training.
  • Exemplar Generation G: Class-specific exemplars are generated from ROI visual and positional features, then pooled into normalized prototype embeddings.ROI tokens are selected from class bounding boxes; learnable class tokens and positional embeddings help form the exemplar representation.
  • Exemplar Generation G: A per-class moving average of q_k produces memory-bank prototypes e_k, stabilizing decoder searches and supporting inference when test-image ROIs are unavailable.The prototypes are also intended to help prevent catastrophic forgetting.
  • Exemplar Generation G: Text embeddings t_k are interleaved with prototype embeddings e_k, with sub-sentence attention masks applying self-attention within each class token group.The combined representations enter the encoder-decoder pipeline for text-and-feature-based detection.
  • Exemplar Generation G: The decoder uses contrastive search over hypothesis boxes, while Lcontrast separates class prototypes and Lfeat aligns each prototype with its top decoder proposal.These losses supplement the original DETR objectives.
  • Iterative training strategy: Iterative training reformulates abnormality detection as binary foreground-versus-background learning, mining backgrounds from normal images and false positives.Stage I uses base annotations; later stages refine novel-class detection with minimal positive samples and can append additional feedback-driven stages.

5. Experiments

The experiments describe implementation and evaluation procedures for EM-DETR, including exemplar settings, background selection, classification and localization metrics, and a heart-class ablation. The supplied passages do not report complete quantitative outcomes beyond the stated evaluation setup and ablation caveat.

  • Experimental details: Experiments average results across 5 runs using four 40 GB A100 GPUs, a 0.0008 learning rate, and frozen image and text backbones.The model uses a linear scheduler and a simple 2-head, 4-layer transformer for F.
  • Experimental details: The moving average uses L = 200 exemplars, and localization performance is reported with mAP50.mAP50 denotes mean average precision at 50% IoU.
  • Background selection: Background boxes are selected as two annotations per image quadrant near the foreground location, with random positional and size perturbations.Stage III additionally uses the top 8 misclassified regions as background annotations.
  • Evaluation metrics: Evaluation includes image-level AUC, localization mAP50 at 50% IoU, and distance-based FROC sensitivity.The highest per-image class probability is used for image-level classification scoring.
  • Ablation: With 5 heart training images and 80 images for other classes, adding the Exemplar Generation module G decreases mAP50 for classes overlapping the heart region.The reported explanation is overlapping region boundaries.

6. Experimental Results

EM-DETR delivers strong few-shot detection across anatomy and thoracic-finding tasks, with performance depending on annotation quantity, sample diversity, and background selection. Qualitative analysis also shows that box-level evaluation can underestimate tightly localized detections.

  • Few-shot comparison: With 5 heart-annotated images, EM-DETR boosts mAP50 from 23% to 92% over Grounding DINO.With 15 images, the heart mAP50 reaches 99%.
  • Few-shot comparison: Using 7% of annotations, EM-DETR improves sensitivity by 10–30% points, AUC by 2–13% points, and mAP50 by 4–16% points over Grounding DINO.With 100% of annotations, mAP50 improves by 3–4% points over the baseline.
  • Finding-specific behavior: EM-DETR shows near-SOTA sensitivity and classification for minimal-shot pleural effusion detection, while pneumothorax and lesions perform less strongly.Pleural effusion’s clustered latent distribution and consistent lung-base location are associated with its stronger results.
  • Finding-specific behavior: Increasing pneumothorax annotations from 200 to 400 substantially improves classification and localization, approaching SOTA performance.The results underscore the importance of sample diversity for findings with heterogeneous appearance and location.
  • Ablation studies: With 14% of pneumothorax annotations, EM-DETR reaches near-SOTA RPS performance; Stage III training adds 7–8% sensitivity and 5.3% mAP50 with 50 positives.Background samples also materially affect performance.
  • Qualitative results: Tightly localized detections may be counted as false positives when ground-truth boxes cover broader or multiple adjacent findings, reducing mAP.Several overlapping DETR hypotheses may be valid positives for one finding but are penalized by the evaluation framework.

7. Discussion and Conclusion

EM-DETR achieves near-SOTA chest-radiograph results with fewer than 10% of annotations and is evaluated across proprietary and public datasets. The framework is designed for new disease classes, but memory use and annotation noise remain practical constraints.

  • Conclusion: The framework is structured to accommodate new disease classes with limited retraining and incorporates domain-aware negative sampling for rare-class detection.These capabilities lay groundwork for future continuous learning.
  • Conclusion: Evaluations across multiple thoracic findings on proprietary and public datasets demonstrate the approach’s robustness and practical utility.
  • Limitations: The current implementation is memory-intensive, and its text encoder primarily acts as a pseudo-class identifier rather than providing meaningful linguistic grounding.Future refinements target memory efficiency and finer-grained feature matching.
  • Limitations: Incomplete labeling of co-occurring findings can negatively affect the contrastive loss and degrade performance.Improved annotation completeness is identified as necessary for greater scalability and robustness.

8. Declaration

The authors disclose using AI tools to improve grammar and readability, followed by human review and editing.

  • The authors used AI tools for grammar and readability improvements, then reviewed and edited the content and retained responsibility for it.

Appendix A.

The appendix examines feature embeddings and few-shot reference counts for the heart class, including performance across k-shot settings. These analyses extend the evaluation of representation structure and annotation efficiency.

  • Embedding analysis: A 2D TSNE analysis examines image-backbone embeddings for regions of interest across classes using TorchXRayVision.
  • Few-shot analysis: The appendix varies the number of few-shot reference images for heart detection on the anatomy dataset.
  • Few-shot analysis: The appendix reports mAP@50 for the Heart class across k-shot examples.
Loading 2608.24281v1…