PaperlayerView exact revision on arXiv

Source-linked AI summary

CytoSPM: Open-Vocabulary Cytopathology Detection with Structured Prompt Bank

Wenjie Li, Zishan Xu, Jinyang Huang, Zhengxin Nie, Shichao Kan, Yixiong Liang

arXiv:2609.31314v1cs.CV

TL;DR

Cytopathology detection needs open-vocabulary recognition because its categories are fine-grained, long-tailed, and evolving, while existing detectors are mostly single-domain and closed-set and the field lacks a unified benchmark. The paper introduces PentaCyto and CytoSPM, pairing multi-domain evaluation with decoupled visual representation extraction and structural prompt matching. On PentaCyto, CytoSPM achieves state-of-the-art overall and novel-category detection while maintaining strong base-category performance and efficient inference.

  • Problem

    Cytopathology has fine-grained, long-tailed, evolving categories, but existing detectors are mostly closed-set and the field lacks a unified open-vocabulary benchmark.

    Source: §I · p. 1

  • Method

    PentaCyto provides a five-domain benchmark with structured cytomorphology prompts, while CytoSPM decouples class-agnostic visual feature extraction from class-aware structural prompt matching.

    Source: §I · p. 2

  • Results

    CytoSPM achieves state-of-the-art overall and novel-category detection on PentaCyto while maintaining strong base-category performance and efficient inference.

    Source: §I · p. 2

  • Takeaways & Limitations

    PentaCyto provides a unified evaluation protocol across five cytology domains, and CytoSPM improves novel-category recognition while avoiding heavy cross-modal fusion.

    Source: §VI · p. 8

    CytoSPM relies on biomedical vision–language pretraining, and its generalization to broader microscopy scenarios remains to be studied.

    Source: §VI · p. 8

Abstract

from arXiv · show

Cytopathology detection requires open-vocabulary recognition because cellular categories are fine-grained, long-tailed, and continuously evolving across different organ systems. However, existing cytology detectors are mostly single-domain and closed-set, and there is still no unified benchmark for evaluating open-vocabulary cytopathology detection. We present PentaCyto, a multi-domain benchmark covering cervical, urinary, respiratory, serous fluid, and thyroid cytology, with 24 base categories and 9 held-out novel categories. Each category is associated with structured cytomorphology prompts that describe diagnostic morphological attributes and provide clinically grounded textual knowledge. We further propose CytoSPM, an efficient detector based on a decoupled two-stage design. It first extracts reusable class-agnostic visual representations, and then performs class-aware structural prompt matching with class names and cytomorphology prompts. On PentaCyto, CytoSPM outperforms existing methods in novel-category detection and open-vocabulary detection while maintaining efficient inference.

I. INTRODUCTION

Cytopathology’s fine-grained, long-tailed, evolving categories expose the limits of closed-set detectors, while the field lacks a unified benchmark for open-vocabulary evaluation. The paper introduces PentaCyto and CytoSPM, which combines a multi-domain benchmark with a decoupled detector using structured morphology prompts.

I. INTRODUCTION

  • Closed-set cytopathology detectors recognize only training-time categories, despite fine-grained, long-tailed labels that vary across organ systems and continue to evolve.The field also lacks public open-vocabulary benchmarks, diverse annotated image–text detection data, and efficient methods suited to high-throughput screening.Source: §I · pp. 1–2
  • PentaCyto provides a unified open-vocabulary benchmark spanning five cytology domains, with 24 base categories and a category-disjoint novel pool.Its categories are paired with structured cytomorphology prompts describing diagnostic attributes used by pathologists.Source: §I · p. 2
  • CytoSPM first extracts class-agnostic region-level visual features, then uses structural prompt matching and combines class-name, structural-prototype, and fine-grained-attribute signals for classification.The prompt bank supports category-relevant selection of visual evidence and morphological attributes.Source: §I · p. 2
  • CytoSPM achieves state-of-the-art overall and novel-category detection on PentaCyto while maintaining strong base-category performance and efficient inference.Evaluation spans all five cytopathology domains.Source: §I · p. 2

II. RELATED WORK

Medical cell detectors commonly use closed-set labels and focus on a single organ or data domain. A unified multi-domain benchmark is needed to assess generalization across anatomical sites, diagnostic systems, and unseen categories.

A. Object Detection in Medical Images

  • Existing cell-detection studies are mostly single-organ or single-domain and use closed-set categories, limiting assessment of generalization across sites and unseen categories.The passage cites BGF-YOLO, SOCR-YOLO, and HERO as examples of medical or cell-detection methods.Source: A. Object Detection in Medical Images · pp. 2–3
  • Figure 2 presents bounding-box counts for categories across five cytology domains.The supplied passage identifies the chart’s scope but does not report particular counts or category-level comparisons.Source: Figure 2 · p. 3
  • A unified multi-domain benchmark is lacking for systematically evaluating open-vocabulary cell detection across anatomical sites, diagnostic systems, and unseen categories.Table I describes PentaCyto statistics across five domains and base and novel splits.Source: A. Object Detection in Medical Images · p. 3, Table I · p. 3

B. Open-Vocabulary Object Detection

Open-vocabulary detection has advanced in natural images through region-text matching and efficient vision-language detectors, with recent work extending it to medical imaging. The paper identifies fine-grained, long-tailed, cross-domain cytopathology as an underexplored setting and introduces an efficient detector for unseen categories.

B. Open-Vocabulary Object Detection

  • Natural-image methods frame open-vocabulary detection as region-text matching or phrase grounding, including GLIP, Grounding DINO, and efficient YOLO-World.YOLO-World combines YOLO-style architectures with vision-language representations for real-time inference and practical downstream deployment.Source: B. Open-Vocabulary Object Detection · p. 3
  • MedROV extends open-vocabulary detection to medical imaging with a large-scale multimodal detection dataset and medical vision-language foundation models.Its stated aim is to improve detection of known and unseen medical structures.Source: B. Open-Vocabulary Object Detection · p. 3
  • Existing medical open-vocabulary studies give limited attention to fine-grained, long-tailed, cross-domain cytopathology scenarios.The proposed framework targets unseen categories in real-world cytology screening while maintaining detection performance and inference efficiency.Source: B. Open-Vocabulary Object Detection · p. 3

III. PENTACYTO DATASET

PentaCyto addresses the lack of a unified multi-domain benchmark for open-vocabulary cytopathology detection. CytoSPM pairs reusable class-agnostic visual features with class-aware matching of class names and cytomorphology prompts.

III. PENTACYTO DATASET

  • PentaCyto addresses the absence of a unified benchmark for multi-domain, cell-level open-vocabulary cytopathology detection.Existing related work spans medical open-vocabulary detection, cytology foundation models, and single-domain cell detection, but none provides that benchmark.Source: §III · p. 3
  • Eight medical experts annotated and verified bounding boxes and categories, with cross-review intended to reduce missed annotations, incorrect labels, and fine-grained category confusion.Categories follow the relevant clinical reporting systems: PSC, TBS, Bethesda, Paris, and TIS.Source: §III · p. 4
  • PentaCyto contains 24 base and 9 novel categories across five cytology domains, with all novel categories and images excluded from training.The novel pool contains 11,462 images and 29,212 cell bounding-box annotations.Source: §III · p. 4

A. Overview

  • A cytomorphology prompt bank complements class names with fine-grained morphological semantics, while mutual selection forms image-conditioned class prototypes for matching.The final score combines class-name, structural prototype, and fine-grained attribute matching.Source: A. Overview · p. 4, Figure 3 · p. 4
  • CytoSPM first extracts class-agnostic visual region representations, then matches class names and structured morphology prompts to those regions.The two-stage design keeps visual feature extraction separate from class-aware structural detection.Source: A. Overview · p. 4

B. Stage I: Class-agnostic Visual Feature Extraction

Stage I uses only the visual branch to produce class-agnostic region features that can be reused for category matching.

B. Stage I: Class-agnostic Visual Feature Extraction

  • The accompanying PentaCyto comparison distinguishes YOLOE variants using textual prompts, visual prompts, or both.The caption says best and second-best results are marked in bold and underlined.Source: Table III · p. 5
  • The input cytology image is processed by a ConvNeXt-T visual backbone, followed by the neck and detection head to produce region features.This stage introduces no class names, textual prompts, or cross-modal fusion.Source: B. Stage I: Class-agnostic Visual… · pp. 4–5
  • The resulting region features are class-agnostic and reusable for matching with arbitrary categories in the subsequent stage.Each vi denotes a region feature; N is the number of regions and D the embedding dimension.Source: B. Stage I: Class-agnostic Visual… · p. 5

C. Stage II: Class-aware Structural Prompt Matching

CytoSPM supplements coarse class-name semantics with structured morphology prompts and matches them to visual regions. Mutual selection retains visually supported evidence and prompts for an image-conditioned structural prototype.

C. Stage II: Class-aware Structural Prompt Matching

  • Class names provide coarse semantics, so CytoSPM adds class-specific morphology prompts encoded with the frozen BioMedCLIP text encoder.The prompts describe structural morphological features that can distinguish visually similar cytopathology categories.Source: C. Stage II: Class-aware Structural… · pp. 5–6
  • The prompt bank is a fixed buffer with no additional learnable class-dimension parameters, and novel-class evaluation replaces embeddings and prompts while detector weights stay unchanged.Source: C. Stage II: Class-aware Structural… · p. 6
  • CytoSPM selects Top-P visual regions using their maximum response to a class's prompt bank, then uses those regions to identify the most relevant morphology prompts.Prompt support is computed from selected class-related visual regions.Source: C. Stage II: Class-aware Structural… · p. 6
  • Mutual selection retains visually supported prompts to reduce noise from irrelevant attributes, and aggregates selected prompts into an image-conditioned prototype representing visually supported fine-grained morphology.Source: C. Stage II: Class-aware Structural… · p. 6

D. Final Matching Score

The final classification score combines class-name, structural-prototype, and fine-grained attribute matching. These signals capture class semantics, image-conditioned structural consistency, and the strongest response to a relevant morphology prompt.

D. Final Matching Score

  • The final classification score is jointly determined by class-name, structural-prototype, and fine-grained attribute matching.Source: D. Final Matching Score · p. 6
  • Class-name matching preserves the basic semantic information supplied by the class name.Source: D. Final Matching Score · p. 6
  • Structural-prototype matching measures visual-region consistency with the image-conditioned prototype, while attribute matching uses the maximum response to the class's most relevant morphology prompt.Source: D. Final Matching Score · p. 6

E. Warm-up for Structural Matching

CytoSPM delays structural prompt matching until visual representations are more stable, reducing noisy supervision during early base-class training. It gradually strengthens structural matching during training and uses the complete score at inference.

E. Warm-up for Structural Matching

A. Experimental Setup

CytoSPM is evaluated on PentaCyto using held-out novel categories, and the reported results show stronger novel-category detection alongside efficient inference. The experiments also compare prompt selection and text encoders.

A. Experimental Setup

  • Top-Q selects morphology prompts, while Top-P selects visual evidence for structural matching.Figure 5 presents their ablation.Source: Figure 5 · p. 7
  • Novel classes are evaluated zero-shot by replacing only class-name embeddings and prompt banks; performance is reported using mAP, mAP50, and mAP75.PentaCyto spans five cytology domains; training uses a frozen BiomedCLIP text encoder and a fixed prompt bank.Source: A. Experimental Setup · pp. 6–7

B. Main Results

  • 15.00 AP, 21.00 AP50, and 17.20 AP75 on novel categories outperform YOLO-World-L by 1.30, 2.70, and 1.40, respectively; CytoSPM reaches 25.8 FPS.CytoSPM also achieves the highest APAVG, 24.27, versus 23.34 for WeDetect.Source: B. Main Results · p. 7
  • BiomedCLIP and CLIP perform comparably on base categories, while BiomedCLIP performs clearly better on novel categories.The comparison is shown in Figure 7.Source: Figure 7 · p. 7
  • CytoSPM more accurately localizes cells in challenging scenes and transfers to a held-out novel thyroid malignancy in qualitative examples.The comparisons include cluttered backgrounds and morphologically similar cells.Source: B. Main Results · p. 7

C. Ablation Studies

The ablations examine matching signals, selection settings, warm-up, and text encoders. The conclusion presents PentaCyto and CytoSPM while noting that broader microscopy generalization remains to be studied.

C. Ablation Studies

  • Adding fine-grained attribute matching to class-name and structural-prototype matching yields the best Novel AP, AP50, and AP75.The authors attribute the improvement to matching visual regions with relevant morphology prompts.Source: C. Ablation Studies · p. 7
  • Q = 0.95 gives the best novel-category results, while P = 0.01 gives the best novel performance among tested visual-region selection settings.Keeping all prompts slightly improves base-category performance but not novel-category results; selecting too few or too many regions weakens alignment.Source: C. Ablation Studies · pp. 7–8
  • Structural-matching warm-up raises novel AP from 14.50 to 15.00 and also raises AP50 and AP75 to 21.00 and 17.20.Without warm-up, base-category performance is slightly better.Source: C. Ablation Studies · p. 8
  • BiomedCLIP and CLIP perform similarly on base categories, but BiomedCLIP consistently outperforms CLIP on novel categories, especially Novel AP50 and AP75.The authors note BiomedCLIP’s medical focus and medical image–text pretraining.Source: C. Ablation Studies · p. 8

VI. CONCLUSION AND LIMITATIONS

  • CytoSPM relies on biomedical vision–language pretraining, and generalization to broader microscopy scenarios remains to be studied.The conclusion also describes PentaCyto as a five-domain evaluation benchmark.Source: §VI · p. 8

Related papers

Loading 2609.31314v1…