Medal S: Spatio-Textual Prompt Model for Medical Segmentation

1 Introduction

• Medical-image segmentation models face challenges from modality and anatomical diversity, while natural-image foundation models do not transfer directly to medical volumes.The introduction notes SAM and SAM 2 as examples of natural-image promptable models.Source: §1 · p. 2
• Downsampling spatial prompts can lose voxel-level detail, and sequential class processing reduces efficiency and limits learning across related anatomical regions.The introduction contrasts these approaches with native-resolution prompting.Source: §1 · p. 3
• Text-only approaches can lack 3D spatial context and support for iterative refinement, while native-resolution spatial and textual prompting had not been explored together for medical segmentation.The introduction also describes CAT as using cropped regions after downsampling.Source: §1 · p. 3
• Medal S combines native-resolution volumetric prompts with text embeddings through channel-wise alignment and lightweight 3D convolution for simultaneous refinement.This design targets misalignment between spatial prompts and text embeddings.Source: §1 · p. 4
• For multi-class segmentation, the approach maintains full fidelity and provides more than 10× faster inference on 24 classes than sequential processing.Dynamic resampling addresses target-patch ratio imbalance, building upon SAT.Source: §1 · p. 4
• The framework supports up to 243 classes across CT, MRI, PET, ultrasound, and microscopy, with text-only self-refinement and hybrid manual-annotation modes.The introduction also cites optimized text preprocessing, two-stage inference, and post-processing.Source: §1 · p. 4

2 Method

• The framework fuses multi-scale image features and text embeddings, then aligns parallel native-resolution spatial prompts with the resulting spatio-textual features for iterative refinement.Spatial prompts may be simulated, predicted, or annotated; for 24-class segmentation, the caption reports more than 10× speedup over sequential processing.Source: §2 · p. 4, Figure 2 · p. 5

2.1 Prompt Encoder

• The prompt encoder aggregates parallel spatial prompts from previous predictions or user annotations through binary thresholding to form a foreground spatial prompt.The caption also describes spatial prompts as being aligned through channel-wise matching while maintaining full fidelity.Source: Foreground spatial prompt encoding · p. 4, Figure 2 · p. 5
• Textual prompt encoding uses a frozen SAT text encoder to process medical terminology prompts and produce embeddings for anatomical targets.The prompt encoder's spatial and textual components are inspired by nnInteractive [6] and SAT, respectively.Source: §2.1 · p. 4, Textual prompt encoding · p. 5

2.2 Spatial Prompt Generation

• Spatial prompt generation produces a single-channel global foreground prompt and a multi-channel class-specific prompt from ground-truth masks.The generator is intended to create realistic coarse segmentations for interactive use and autonomous refinement.Source: §2.2 · pp. 5–6
• Controlled stochastic transformations simulate empty, incomplete, added, and channel-dropped prompts through block operations and channel-level variation.The process applies mutually exclusive drop and add masks, then binarizes both prompt types.Source: §2.2 · p. 6
• The generated prompts span coarse segmentations to imperfect annotations, supporting generalization across varying prompt quality while maintaining anatomical plausibility.The transformations are intended to represent real-world variation in prompt quality.Source: §2.2 · p. 6

2.3 Query Decoder

• The decoder preserves complete 3D spatial context in volumetric prompts rather than compressing them into low-dimensional mask embeddings.The paper identifies native spatial resolution as important for 3D medical imaging.Source: §2.3 · p. 6
• Multiple native-resolution masks can be processed in parallel, improving multi-class segmentation performance.The decoder’s channel-wise alignment is described as mitigating accuracy degradation from resolution mismatches.Source: §2.3 · p. 6
• The decoder adapts text embeddings using multi-scale visual features, with semantic queries corresponding to anatomical targets.Its voxel features are derived by progressively upsampling visual encoder outputs with U-Net-style skip connections.Source: §2.3 · p. 7
• Adapted text embeddings are aligned with volumetric prompts, then concatenated with original voxel features and processed by a lightweight 3D convolutional module.The module jointly optimizes voxel-space features while maintaining channel-wise alignment.Source: §2.3 · pp. 7–8
• The final prediction is a voxelwise probability map with dedicated channels for anatomical structures and pathological regions.Voxelwise query-feature correlation is followed by sigmoid activation.Source: §2.3 · p. 8

2.4 Iterative Inference

• Each iteration uses the preceding prediction as the spatial prompt for the next, progressively refining segmentation.Complementary masked predictions are aggregated to improve robustness against input noise.Source: §2.4 · p. 8
• Iterations enhance features through text-query and spatial-prompt cross-attention, predict masked and unmasked regions in parallel, and average predictions across rounds.The described block-mask sizes are 4 and 8.Source: §2.4 · p. 8
• Text-only inference starts from zero prompts and self-refines, while hybrid inference can incorporate external spatial cues such as manual annotations.The inference pipeline dynamically updates spatial prompts during refinement.Source: §2.4 · p. 8

2.5 Dynamic Resampling

• Dynamic resampling adjusts voxel spacing using the smallest foreground component or class-specific target as a reference for the fixed patch size.The approach addresses false positives when targets exceed the patch and false negatives when targets are much smaller.Source: §2.5 · p. 9
• The smallest target determines resampling parameters, rather than assigning each target its own rate, to balance representation with computational efficiency.The adjusted image spacing is chosen so the target’s physical dimensions align with the patch size.Source: §2.5 · p. 9
• Adjusted spacing is bounded by practical constraints that depend on target class and inference stage.The passage identifies adjusted spacing and image dimensions as components of the resampling formula.Source: §2.5 · p. 10

2.6 Two-stage Inference

• A coarse-to-fine strategy uses low-resolution segmentation to locate regions of interest before high-resolution refinement.The approach is intended for localized targets and reduces inference time compared with full high-resolution processing on datasets with small foreground regions.Source: §2.6 · p. 10
• The coarse stage uses 1.5 × 1.5 × 3.0 voxel spacing and a 224 × 224 × 128 crop; if it finds no foreground, inference defaults to the full volume at high resolution.The high-resolution model uses 1.0 × 1.0 × 1.0 spacing.Source: §2.6 · p. 10
• The second stage expands the coarse prediction’s ROI by a factor of 1.1 to 1.5, resamples it at 1.0 × 1.0 × 1.0 spacing, and constrains memory use below 32 GB.It uses a 192 × 192 × 192 crop and refines with coarse predictions as spatial prompts.Source: §2.6 · p. 10

2.7 Text Prompts Preprocessing

• A modality-specific class mapping assigns unique identifiers and canonical names to structures and lesions across CT, MRI, ultrasound, PET, and microscopy prompts.Names are standardized, for example, “Left renal structure” to “Left kidney” and “Myocardium” to “Heart.”Source: §2.7 · p. 11
• A variant mapping handles anatomical and lesion terminology, directionality, and suffixes, prioritizing longer terms to avoid partial matches.For example, “hepatic lesions” maps to “Liver lesions,” while “Brainstem” is distinguished from “Brain.”Source: §2.7 · p. 11
• Prompt parsing extracts modality, class identifier, and canonical name using the mappings, with longer matches and directional-pattern checks improving specificity.The instance label distinguishes anatomy from lesion prompts.Source: §2.7 · p. 11
• The extracted modality and class identifier feed a text encoder and guide modality-aware processing, including class-specific resampling and segmentation refinements.The text embeddings guide dynamic resampling and post-processing.Source: §2.7 · p. 11

2.8 Loss Function

• The training loss adds binary cross-entropy and Dice terms to optimize pixel-level accuracy and region overlap.The formulation is stated for N classes and C voxels.Source: §2.8 · pp. 11–12

2.9 Post-processing

• Post-processing selects components independently for each class, using component probability and size to refine masks beyond nnU-Net’s single largest-component operation.The method is described as suppressing spurious predictions while preserving anatomically plausible structures.Source: §2.9 · p. 12
• Among each class’s three largest components, the method retains those with mean probabilities above 0.86 and within 0.1 of the maximum.If none qualify, it applies a fallback based on the highest-probability component’s rank and relative size, otherwise selecting the largest component.Source: §2.9 · p. 12
• Class-wise probability-guided component selection is described as improving multi-class segmentation robustness.Source: §2.9 · p. 12

3.1 Data and Evaluation Methodology

• The development set extends the MedSAM on Laptop Challenge with additional 3D cases, while the hidden test set contains previously unpublished cases across CT, MRI, PET, ultrasound, and microscopy.Annotations were supplied by contributors or generated by organizers; the challenge also offers a coreset track using 10% of training cases.Source: §3.1 · pp. 12–13
• Evaluation measures semantic segmentation with DSC and NSD and instance segmentation with F1 at 0.5 overlap and DSC for correctly identified instances.A 60-second-per-class runtime limit applies; exceeding it zeros DSC and NSD scores for affected test cases.Source: §3.1 · p. 13

3.2 Implementation Details

• Images are converted to npz and normalized to [0, 255], using modality-specific preprocessing and standard CT windows before rescaling.Non-CT intensities are truncated at the 0.5th and 99.5th percentiles; images already in range need no further preprocessing.Source: Data preprocessing · p. 14
• Training uses an nnU-Net-based dynamic resampling workflow, CPU resampling, Blosc2 compression, and modified nnU-Net preprocessing, loading, and augmentation code.Text dataloading uses dataset-specific sampling rates, positive-negative ratios, and padding alignment; multi-GPU training is primarily based on SAT.Source: Training Protocols · pp. 14–15
• Training uses effective batch sizes of 4 or 8 for the two stated input sizes and takes 168 hours to complete.The configurations are 224×224×128 with batch size 2 per GPU across 2 GPUs, and 192×192×192 with batch size 2 per GPU across 4 GPUs.Source: Training Protocols · p. 15

4.1 Quantitative Results on Validation Set

• On validation, Medal S averages 75.44 DSC, 77.34 NSD, 38.24 F1, and 65.46 DSC TP, with gains over CAT of 4.71/4.74/23.79/33.12 and over SAT of 5.61/6.28/13.36/18.49.It falls below BiomedParse-V in semantic segmentation but achieves comparable performance in instance segmentation.Source: §4.1 · pp. 15, 17, Table 4 · p. 16

4.5 Limitation and future work

• Ultrasound remains a resampling challenge: Medal S scores 82.45 DSC versus SAT’s 85.58 DSC, with large target-to-patch ratios cited as a cause.Source: §4.1 · p. 17

4.2 Qualitative results on validation set

• Across five modalities, validation examples show effective multi-class segmentation and boundary delineation for substantial-volume regions, while smaller lesions, imbalance, and blurred boundaries remain challenging.The qualitative comparison includes both good and bad cases against ground truth; label noise contributes to difficulty, and resampling and preprocessing mitigate some issues.Source: §4.2 · pp. 17–18, Figure 3 · p. 18

4.3 Spatial Prompt Ablation Study

• With 24 classes, parallel prompting runs in 40.63s and uses 12.5GB versus 435.1s for sequential prompting; its runtime and memory are 31.75s and 9.49GB without prompts.The comparison is for one test case; parallel spatial prompting also yields 83.98 DSC and 79.16 NSD in Table 5.Source: Figure 4 · p. 19, §4.3 · pp. 19, 21, Table 5 · p. 19
• Using ground-truth masks as spatial prompts yields the highest DSC and NSD in the prompt ablation, with qualitative improvements in noise suppression, category-confusion resolution, and spatial continuity.Source: §4.3 · pp. 18–19, Table 5 · p. 19, Figure 5 · p. 20

4.4 Results on final testing set

• On the all-data test set, Medal S achieves 58.06 Dice and 59.11 NSD, exceeding SAT’s 54.13 DSC and 52.97 NSD but trailing BiomedParse-V’s 74.97 DSC and 77.47 NSD.Source: Table 6 · p. 21, §4.4 · p. 21

4.5 Limitation and future work

• Dynamic resampling needs further refinement for ultrasound’s large targets, while future work also targets small lesions, foreground-background imbalance, blurred boundaries, and datasets with many classes and intricate anatomy.Source: §4.5 · pp. 21–22

• Parallel spatial prompting yields a 10× speedup for 24-class segmentation over sequential methods, with greater efficiency gains for more classes.Source: §5 · p. 22
• Medal S supports native-resolution spatial and textual prompts in an end-to-end trainable framework, aligning prompt channels with text embeddings to mitigate resolution-mismatch inaccuracies.Source: §5 · p. 22
• A lightweight 3D convolutional module enables precise voxel-space refinement guided by both prompt types.Source: §5 · p. 22
• On the five-modality validation average, Medal S outperforms SAT with DSC 75.44 (vs. 69.83), NSD 77.34 (vs. 71.06), F1 38.24 (vs. 24.88), and DSC TP 65.46 (vs. 46.97).Source: §5 · p. 22
• The model supports up to 243 classes across five modalities and offers text-only self-refinement or hybrid prompting with manual annotations.Source: §5 · p. 22
• Future work targets robustness on complex, imbalanced datasets and small lesions, and dynamic-resampling improvements for ultrasound performance.Source: §5 · p. 22