Source-linked AI summary
BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once
Theodore Zhao, Yu Gu, Jianwei Yang, Naoto Usuyama, Ho Hin Lee, Tristan Naumann, Jianfeng Gao, Angela Crabtree, Jacob Abel, Christine Moung-Wen, Brian Piening, Carlo Bifulco, Mu Wei, Hoifung Poon, Sheng Wang
TL;DR
Biomedical image analysis often separates segmentation, detection, and recognition and relies on labor-intensive bounding boxes. BiomedParse jointly learns these tasks using harmonized biomedical text supervision, achieving state-of-the-art performance across modalities while enabling text-prompted and holistic image parsing. Its current limitations include instance separation, conversational interaction, and support for non-2D modalities.
Problem
Existing biomedical methods often focus on segmentation alone and require tight bounding boxes, limiting semantic integration and scalability, especially for irregular objects.
Method
BiomedParse jointly learns segmentation, detection, and recognition using a GPT-4-harmonized ontology and text descriptions derived from biomedical segmentation datasets.
Results
BiomedParse achieved state-of-the-art segmentation, detection, and recognition performance across 102,855 test triples spanning 9 imaging modalities, including stronger gains for irregular-shaped objects.
Takeaways & Limitations
BiomedParse enables text-prompted segmentation without bounding boxes and recognition that segments and labels all biomedical objects in an image.
Takeaways & Limitations
BiomedParse does not yet differentiate individual object instances by default, lacks conversational interaction, and does not fully support non-2D modalities.
Abstract
from arXiv · showhide
Biomedical image analysis is fundamental for biomedical discovery in cell biology, pathology, radiology, and many other biomedical domains. Holistic image analysis comprises interdependent subtasks such as segmentation, detection, and recognition of relevant objects. Here, we propose BiomedParse, a biomedical foundation model for imaging parsing that can jointly conduct segmentation, detection, and recognition for 82 object types across 9 imaging modalities. Through joint learning, we can improve accuracy for individual tasks and enable novel applications such as segmenting all relevant objects in an image through a text prompt, rather than requiring users to laboriously specify the bounding box for each object. We leveraged readily available natural-language labels or descriptions accompanying those datasets and use GPT-4 to harmonize the noisy, unstructured text information with established biomedical object ontologies. We created a large dataset comprising over six million triples of image, segmentation mask, and textual description. On image segmentation, we showed that BiomedParse is broadly applicable, outperforming state-of-the-art methods on 102,855 test image-mask-label triples across 9 imaging modalities (everything). On object detection, which aims to locate a specific object of interest, BiomedParse again attained state-of-the-art performance, especially on objects with irregular shapes (everywhere). On object recognition, which aims to identify all objects in a given image along with their semantic types, we showed that BiomedParse can simultaneously segment and label all biomedical objects in an image (all at once). In summary, BiomedParse is an all-in-one tool for biomedical image analysis by jointly solving segmentation, detection, and recognition for all major biomedical image modalities, paving the path for efficient and accurate image-based biomedical discovery.
Introduction
BiomedParse treats biomedical image analysis as a unified parsing problem spanning segmentation, detection, and recognition. It combines these tasks to reduce bounding-box burden and improve analysis across biomedical images.
- Motivation: Biomedical image analysis comprises interdependent segmentation, detection, and recognition tasks that standard methods typically handle separately.Segmentation divides images into object regions, detection locates objects, and recognition identifies all objects.
- Motivation: Segmentation-focused methods can ignore semantic information from detection and recognition while requiring users to draw tight bounding boxes.This creates substantial manual burden and is difficult for users without domain expertise.
- Approach: BiomedParse jointly learns segmentation, detection, and recognition so text prompts can specify objects without user-provided bounding boxes.The model uses semantic labels from text prompts to guide segmentation.
- Data: GPT-4 harmonizes noisy natural-language labels and descriptions from segmentation datasets with established biomedical object ontologies.This addresses the scarcity of datasets covering all three tasks using standard segmentation datasets.
- Capabilities: BiomedParse models typical object shapes, segments irregular objects from text prompts alone, and recognizes all objects without a user prompt.These capabilities target limitations of rectangular bounding-box-based methods.
- Evaluation: 102,855 held-out image-mask-label triples across 9 modalities were used for evaluation, with BiomedParse achieving new state-of-the-art segmentation results.It outperformed prior best methods including MedSAM and SAM.
Results
BiomedParseData combines biomedical segmentation datasets with harmonized semantic labels and descriptions to support joint image parsing. Across segmentation, invalid-prompt detection, irregular-object detection, and recognition, BiomedParse shows strong performance and broad applicability.
- Dataset: BiomedParseData combines 45 biomedical image segmentation datasets and uses GPT-4 to generate canonical semantic labels for segmented objects.The dataset was created because no prior dataset was known to cover segmentation, detection, and recognition together.
- Dataset: GPT-4 harmonizes noisy natural-language object descriptions with a unifying biomedical ontology spanning histology, organs, abnormalities, and specific object types.This converts inconsistent dataset text into semantic supervision for image parsing.
- Dataset: 3.4 million image-mask-label triples span 9 imaging modalities and 25 anatomic sites.The resulting dataset provides diverse semantic-based biomedical image-analysis data.
- Dataset: 6.8 million image-mask-description triples were obtained by synthesizing synonymous text descriptions for semantic labels.These descriptions broaden the text prompts represented during training.
- Model: BiomedParse uses an image encoder, text encoder, mask decoder, and meta-object classifier in a modular SEEM-based design.The image and text encoders were initialized with Focal and PubMedBERT, respectively.
- Representation: BiomedParse text embeddings distinguished fine-grained cell types with a 0.89 Silhouette score, exceeding PubMedBERT embeddings.The embeddings were also more predictive of tumor malignancy on a pathology dataset.
- Segmentation: 102,855 test instances across 9 modalities produced the best Dice score in segmentation, including against MedSAM with oracle bounding boxes.BiomedParse was especially advantageous in pathology and CT, where irregular-shaped objects are common.
- Segmentation: A single colon-pathology text prompt yielded a 0.942 median Dice score, while MedSAM required tight boxes for 430 cells to achieve comparable results.Neither SAM nor MedSAM exceeded 0.75 median Dice without tight bounding boxes in this comparison.
Discussion
BiomedParse unifies biomedical segmentation, detection, and recognition through joint learning and text prompting. It achieves broad performance gains while retaining important scope boundaries for instance separation, conversational interaction, and native 3D analysis.
- Unified image parsing: BiomedParse combines segmentation, detection, and recognition in one bounding-box-free image-parsing framework.The model uses joint learning and semantic labels to support holistic biomedical image analysis.
- Evaluation: 102,855 held-out image-mask-label triples across 9 modalities supported large-scale evaluation, with BiomedParse outperforming prior best methods such as MedSAM and SAM.The comparison remained favorable even when competing methods received oracle bounding boxes.
- Practical capabilities: BiomedParse enables text-prompted segmentation and recognition of all objects in an image, addressing labor-intensive bounding-box workflows.Its joint framework also targets irregular shapes and supports scaling holistic image analysis.
- Cellular imaging: In cellular imaging, BiomedParse can combine cell segmentation and identification across diverse modalities and cell types.This extends prior work that often relies on bounding-box detection and struggles with diverse cell morphologies.
- Limitations: BiomedParse currently requires post-processing for individual instances, lacks conversational interaction, and processes CT and MRI as 2D slices rather than full 3D data.These limitations define current boundaries for cell counting, complex user interaction, and volumetric or temporal information use.
Methods
The methods construct a multi-source biomedical parsing dataset, harmonize object language with GPT-4, and use modular text-conditioned prediction. Additional procedures address invalid prompts, shape visualization, irregularity measurement, and holistic object assignment.
- Dataset construction: BiomedParseData synthesizes 45 public segmentation datasets into image-level object collections with masks, canonical ontology labels, and synonymous descriptions.Each image is associated with multiple annotated objects and textual variants for training.
- Evaluation protocol: Training and testing use an 80%/20% split within each dataset, while slices from each 3D volume remain in the same split to prevent leakage.This split rule applies to model training and evaluation.
- Ontology and text normalization: GPT-4 organizes biomedical descriptions into a three-layer ontology containing 3 broad categories, 15 meta-object types, and 82 fine-grained object types.The ontology harmonizes noisy language and supports semantic supervision.
- Model design: The model uses an image encoder, text encoder, mask decoder, and meta-object classifier to map an image and text prompt to object masks and semantic types.The mask decoder produces per-pixel probabilities at the original image size.
- Invalid-prompt detection: Invalid text prompts are evaluated with object-specific probability distributions and one-sample K-S-test p-values, alongside Grounding DINO confidence scores.Invalid examples are sampled by assigning nonexistent object types to images.
- Shape visualization: Shape attention maps are built by aggregating and cross-correlation-aligning predicted pixel attentions across examples, with volume-level aggregation for CT and MRI.The procedure estimates typical spatial shape for each target type.
- Irregularity metrics: Target regularity is quantified using Box Ratio, Convex Ratio, and inverse rotational inertia, with round masks attaining IRI equal to 1.These metrics characterize tight-box similarity, convexity, and spatial spread.
- Holistic recognition: For holistic recognition, the model sequentially predicts supported targets, assigns overlapping pixels to the highest-probability target, and selects targets exceeding a user-defined area threshold.This produces final target areas after pixel-wise competition.
Data availability
The authors planned to provide BiomedParseData or scripts for reproducing it from the original datasets upon publication.
- Data availability: BiomedParseData or reproduction scripts will be provided upon publication.The planned access covers either the dataset or scripts that recreate it from source datasets.
Code availability
The authors planned full release of BiomedParse resources and replication guidance upon publication.
- Code availability: Model weights, source code, and detailed implementation steps for pre-training, fine-tuning, and inference will be made available upon publication.The stated goal is to facilitate independent replication.
Pathologist annotation BiomedParse annotation
Figure 5 compares pathologist annotations with BiomedParse outputs on de-identified pathology images from Providence Health System. The comparison includes object names and locations, showing BiomedParse’s annotation outputs alongside pathologist results.
- Real-world pathology comparison: BiomedParse outputs are compared directly with pathologist annotations on six de-identified pathology examples.The figure pairs pathologist annotations in panels a, c, and e with BiomedParse annotations in panels b, d, and f.
- Annotation content: The comparison displays object names, such as lymphocyte and stroma, together with their object locations.The figure reports the exact pathologist outputs and the exact BiomedParse outputs.
- Input-free cell annotation: BiomedParse identifies and segments cells of any types included in its ontology without user-provided inputs.This capability is presented in the real-world pathology examples.