PaperlayerView exact revision on arXiv

Source-linked AI summary

SegBanana: Steering Unified Multimodal Models into Medical Segmenters

Xiaoye Liang, Ye Yan, Mingze Yin, Shikun Feng, Mai Xu, Haiguang Liu, Lai Jiang, Yiheng Zhu

arXiv:2609.34235v1cs.CV

TL;DR

Medical segmentation systems can generalize poorly under clinical distribution shifts, while pixel-level annotations for adaptation are often scarce and costly. SegBanana uses a frozen UMM with retrieval, critique, and stateful iterative generation and refinement; across eight datasets, it achieves a 77.45% average mDice and remains robust to out-of-domain visual supports.

  • Problem

    Medical segmentation models can generalize poorly beyond their training distributions, while the pixel-level annotations needed for adaptation are often scarce and costly.

    Source: §1 · p. 1

  • Method

    SegBanana uses a frozen UMM with anatomy-aware retrieval, comparative critique, and a state-aware controller to iteratively generate and refine masks without task-specific post-training.

    Source: §1 · pp. 2–3

  • Results

    Across eight datasets, SegBanana achieves an average mDice of 77.45%, outperforming BiomedParse by 14.93 points and SegGPT by 25.19 points.

    Source: §1 · p. 3, §5.1 · p. 7

  • Takeaways & Limitations

    SegBanana demonstrates training-free transfer across diverse clinical scenarios and robustness when only out-of-domain visual supports are available.

    Source: §1 · p. 3, §6 · p. 9

    Comparative Quality Critique does not reliably select the best generated candidate, so selected predictions remain below oracle performance across iterations.

    Source: §5.3 · pp. 8–9

Abstract

from arXiv · show

Medical image segmentation remains challenging in practical deployment, as models often struggle to generalize beyond the distributions covered by their training data and high-quality pixel-level annotations are typically unavailable for adaptation. Inspired by the cross-task transferability of large language models, we investigate whether unified multimodal models (UMMs) can transfer their pretrained visual understanding, reasoning, and generation capabilities to medical image segmentation without task-specific post-training. By recasting segmentation as structured visual generation, we find that frontier UMMs (e.g., Nano Banana) already exhibit basic segmentation capabilities across diverse clinical scenarios, but still struggle with challenging tasks requiring specialized anatomical or domain-specific knowledge. We further show that these limitations can be effectively mitigated by incorporating visual anatomical knowledge from in-context exemplars, expanding candidate solutions through repeated sampling, and refining suboptimal predictions via targeted editing.Motivated by these observations, we propose SegBanana, to our knowledge, the first agentic visual generation framework for training-free medical image segmentation. SegBanana builds on a frozen UMM as the core generative model, augmented with Anatomy-Aware Knowledge Retrieval and Comparative Quality Critique to unlock its potential segmentation capability. A State-Aware Multimodal Controller maintains structured state and iteratively orchestrates these tools, repeatedly refining intermediate predictions toward higher-quality masks. Across eight medical segmentation datasets, SegBanana achieves an average mDice of 77.45%, outperforming representative generalist (SAM3 and SegGPT) and medical-specific (BiomedParse and MedSAM3) baselines by at least 14.93 points, while remaining robust to out-of-domain visual supports.

1 INTRODUCTION

Medical segmentation must generalize across clinical shifts despite scarce pixel-level annotations, motivating approaches that transfer without task-specific supervision. The authors find that unified multimodal models have basic zero-shot capability, then introduce SegBanana to extend it through stateful, feedback-driven inference.

1 INTRODUCTION

  • Distribution shifts from devices, protocols, sites, populations, or disease presentations challenge deployment, while pixel-level annotations for adaptation are scarce and costly.These constraints motivate methods that generalize to new clinical scenarios with limited task-specific supervision.Source: §1 · p. 1
  • Existing approaches rely on capabilities learned from medical training data or pretrained tools, or on correspondence with annotated supports, motivating more flexible transfer.Medical-specific and agentic methods may degrade beyond their coverage; generalist open-vocabulary and in-context methods can lack anatomical knowledge or depend on well-matched supports.Source: §1 · pp. 1–2
  • UMMs show basic zero-shot medical segmentation, but struggle when accurate masks require fine-grained discrimination of adjacent or nested anatomy.Visual anatomical exemplars, repeated sampling, and targeted editing can address different parts of this gap; editing helps when correction guidance is informative.Source: §1 · p. 2
  • SegBanana combines a frozen UMM with anatomy-aware retrieval, mask generation and editing, and comparative critique, orchestrated by a state-aware controller.The controller uses structured interaction state and tool observations to adaptively select actions and iteratively improve masks.Source: §1 · p. 2
  • The study identifies knowledge augmentation, output exploration, and targeted editing as effective mechanisms for training-free transfer.These findings motivate SegBanana's inference-time tool use.Source: §1 · pp. 2–3
  • Experiments across eight datasets show strong generalization across clinical scenarios and robustness when only out-of-domain visual supports are available.The authors report substantial improvement over frozen UMMs.Source: §1 · p. 3

2 RELATED WORK

Prior medical, generalist, and agentic segmentation approaches offer different forms of task adaptation but retain training, tool, knowledge, prompt, or support dependencies. This work instead investigates training-free transfer through reasoning over task semantics and visual context.

Medical Image Segmentation Models

  • Medical segmentation models rely on task-specific training, while recent agentic frameworks depend on trained policies and segmentation tools, limiting adaptation beyond their training distributions.The proposed alternative reasons over task semantics and visual context for training-free adaptation to unseen clinical scenarios.Source: §2 · p. 3
  • General open-vocabulary methods can be constrained by limited medical knowledge and user prompts, whereas in-context methods remain sensitive to visual demonstrations and feature correspondence.The paper explores intrinsic pretrained UMM capabilities for medical segmentation through agentic inference alone, rather than task-level supervision and post-training.Source: §2 · p. 3

Agentic Frameworks for Image Generation and Editing

  • Agentic image-generation and editing frameworks use collaboration, evaluation, reasoning, and iterative tools to improve control within their original tasks; this paper studies transfer beyond those settings.The target is training-free medical image segmentation using pretrained UMM capabilities.Source: §2 · p. 3

3 OBSERVATION

The authors examine how supports, stochastic sampling, and editing affect UMM segmentation when it is treated as conditional visual generation. Their observations motivate a controller that chooses retrieval, exploration, or refinement according to the evolving prediction state.

Generation Exploration

  • Best-of-N performance improves as sampling increases, and sampling across varied supports outperforms repeated sampling with a fixed support.The authors interpret these results as evidence that stochastic outputs and conditioning diversity expose higher-quality candidate predictions.Source: Figure 2 · p. 4, §3 · p. 4

Editing Behavior

  • More specific correction guidance yields larger editing improvements, and editing is more reliable for region-level errors than recognition-level errors.Editing is therefore better suited to refining plausible predictions than recovering fundamentally incorrect ones.Source: Figure 2 · p. 4, §3 · p. 4

Generation Capability

  • UMMs perform better on tasks with salient global shape or appearance cues than on tasks requiring fine discrimination of adjacent or nested structures.This suggests generic visual cues can support segmentation, but specialized anatomical and structural knowledge remains important for accurate predictions.Source: §3 · p. 4

Generation Conditioning

  • Query-relevant visual supports consistently improve tasks requiring specialized anatomical knowledge, while random supports provide less reliable gains and may introduce irrelevant context.The comparison uses image–mask exemplars, with query-relevant supports retrieved using DINO features.Source: §3 · p. 4

Toward Agentic Inference

  • The observed benefits of retrieval, resampling, and editing motivate a controller that chooses among acquiring knowledge, exploring masks, and refining predictions as state evolves.The controller is intended to adapt its actions to the current segmentation state.Source: §3 · p. 5

4 METHOD

SegBanana casts medical image segmentation as structured visual generation, using a frozen UMM in a feedback-driven loop. Its retrieval, generation, critique, and state-aware control components iteratively explore and refine masks.

4 METHOD

  • SegBanana uses a frozen UMM in a closed loop that retrieves visual supports, generates or edits masks, critiques candidates, and updates state until acceptance or budget exhaustion.The controller selects actions from feedback about candidate preferences and residual errors.Source: §4 · p. 5

4.1 CAPABILITY-ENHANCING SEGMENTATION TOOLSET

  • Anatomy-Aware Knowledge Retrieval uses maximal marginal relevance to balance query relevance against redundancy and return task-relevant visual supports.The supports provide anatomical, semantic, and appearance cues for subsequent generation.Source: §4.1 · pp. 5–6
  • The frozen UMM can GENERATE a mask from the query and instruction or EDIT the retained prediction using targeted correction instructions.These modes explore alternative predictions and locally refine promising results.Source: §4.1 · p. 6
  • Comparative Quality Critique compares candidates, diagnoses residual errors, and returns structured feedback; multi-label tasks can retain different labels from different candidates.If no prediction is retained, candidates are compared sequentially; otherwise, the current best is challenged by new candidates.Source: §4.1 · p. 6

4.2 STATE-AWARE MULTIMODAL CONTROLLER

  • The controller maintains compact current and historical state, then chooses GENERATE, EDIT, or ACCEPT according to the current diagnosis and prior interaction.It preserves the best prediction so weaker generations or unsuccessful edits do not overwrite stronger results, and stops when accepted or the iteration budget is exhausted.Source: §4.2 · pp. 6–7

5 EXPERIMENTS

Across eight medical segmentation datasets, SegBanana achieves the highest average mDice and remains robust to out-of-domain supports. Ablations and case studies highlight the value of its components and adaptive inference, while critique-based candidate selection remains a bottleneck.

5 EXPERIMENTS

  • The evaluation covers eight datasets and compares medical-specific and generalist baselines in a training-free setting.The datasets span modalities and targets including histopathology nuclei, retinal vessels, skin and breast lesions, polyps, cardiac structures, and brain tumors.Source: §5 · p. 7

5.1 MAIN RESULTS

  • 77.45% average mDice; SegBanana exceeds BiomedParse by 14.93 points and SegGPT by 25.19 points across eight datasets.It performs best on four datasets and remains competitive with medical-specific methods on the other four; some methods that outperform it have benchmark exposure during training.Source: §5.1 · p. 7, Table 2 · p. 8
  • SegBanana stays close to the in-domain/out-of-domain mDice diagonal on most datasets, while several baselines degrade substantially.SegGPT drops from 53% to 17% on TNBC and from 45% to 16% on ACDC; the authors connect this robustness to reducing annotation burden in new clinical practice.Source: §5.1 · pp. 7–8, Figure 4 · p. 8

5.2 ABLATION

  • Removing retrieval, the critic, or state context lowers average mDice from 77.45% to 72.52%, 72.33%, or 72.72%, respectively.Retrieval matters particularly on Drishti-GS and ACDC; direct Nano Banana 2 generation reaches 57.69% average mDice.Source: §5.2 · p. 8, Table 3 · p. 9

5.3 DISCUSSION

  • Closed-loop inference expands the pool of strong predictions, but Comparative Quality Critique does not consistently identify the best candidate.The selected prediction remains below the oracle across iterations, making candidate selection a remaining bottleneck.Source: §5.3 · pp. 8–9, Figure 5 · p. 9
  • In a case study, the controller switches from regeneration after over-segmentation to selective cup refinement, yielding a final mDice of 97.1%.It preserves the reliable optic disc once a stronger candidate emerges.Source: §5.3 · p. 9

6 CONCLUSION

SegBanana uses a frozen UMM for medical image segmentation without task-specific post-training, combining inference-time retrieval, refinement, and verification. It outperforms medical-specific and generalist baselines across diverse datasets, but imperfect critique reliability limits its ability to exploit the UMM’s segmentation potential.

6 CONCLUSION

  • SegBanana enables a frozen UMM to perform medical image segmentation without task-specific post-training.It uses retrieval, refinement, and verification at inference time.Source: §6 · p. 9
  • Across diverse datasets, SegBanana outperforms medical-specific and generalist baselines and remains robust without in-domain supports.Source: §6 · p. 9
  • Imperfect comparative quality critique prevents SegBanana from fully exploiting the UMM’s segmentation potential; future work targets stronger verification and more autonomous agentic control.The proposed direction includes a dedicated policy model for reasoning over intermediate outcomes and orchestrating visual actions.Source: §6 · p. 9

AI USE STATEMENT

The statement limits generative and multimodal models to experimental components and describes the use of GPT-family models for language polishing and grammar correction.

AI USE STATEMENT

  • GPT-family models were used only for language polishing and grammar correction; generative and multimodal experimental components are described in the paper.AI tools were not used for literature review or idea formation.Source: AI USE STATEMENT · p. 9

ETHICS STATEMENT

The study uses public medical segmentation datasets without collecting patient data or interacting with patients. It is intended for research, not clinical use, and acknowledges that segmentation errors may still occur.

ETHICS STATEMENT

  • The study uses only publicly available medical image segmentation datasets; it collected no new patient data, recruited no human subjects, and involved no patient interaction.The authors also state that they do not attempt to identify or re-identify individuals from the images.Source: ETHICS STATEMENT · p. 10
  • SegBanana is intended solely for research purposes and is not a medical device.Source: ETHICS STATEMENT · p. 10
  • Segmentation errors may still arise under challenging anatomy, distribution shifts, or imperfect candidate evaluation, and the authors do not claim clinical readiness.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper details implementation settings and plans to release the materials needed to reproduce the experiments. It notes that externally hosted UMM APIs can make exact outputs vary.

REPRODUCIBILITY STATEMENT

  • The authors specify model configurations, prompts and input–output formats, retrieval and state settings, rollout parameters, and inference limits.The specifications include the UMM and controller used, DINOv3-based retrieval and MMR settings, candidates per action, and maximum inference iterations.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Upon publication, the authors will release code, prompt templates, configuration files, dataset splits, support-pool construction, and preprocessing.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Externally hosted UMM APIs may cause exact outputs to vary because of stochastic inference or future API/model updates.The authors plan to document the model versions and inference settings used for reported results.Source: REPRODUCIBILITY STATEMENT · p. 10

A LIMITATIONS AND FUTURE WORK

The appendix states that SegBanana is limited to 2D processing and that its candidate critique is imperfect. It also describes the evaluation datasets, support construction, and inference prompts and actions.

A LIMITATIONS AND FUTURE WORK

  • SegBanana processes 3D scans slice by slice and cannot directly model volumetric 3D context; imperfect critique reliability also limits candidate selection and refinement.The authors identify both as current limitations and suggest targeted supervision as future work.Source: A LIMITATIONS AND FUTURE WORK · p. 15

B.1 DATASET DETAILS AND COVERAGE

  • The eight evaluation datasets span six modalities, varied anatomical targets and spatial scales, and adjacent, nested, and irregular structures.The resulting challenges include subtle boundaries, anatomical coupling, dense clustering, topology preservation, and appearance variation.Source: §B.1 · p. 15

B.2 SUPERVISION PROFILE OF COMPARED METHODS

  • The supervision profile counts datasets used at any model-development stage as exposure and marks benchmark overlap without implying identical samples or splits.For methods with undisclosed pretraining corpora, the paper reports only documented datasets or modality-level exposure and makes no assumptions about undisclosed data.Source: §B.2 · pp. 15–16, Table 5 · p. 16

B.3 OUT-OF-DOMAIN SUPPORT CONSTRUCTION

  • The out-of-domain setting pairs each query dataset with an external annotated dataset sharing the same or a closely corresponding target while differing in acquisition, appearance, population, or annotation.The pairings preserve task-relevant anatomical semantics while introducing domain shifts.Source: §B.3 · pp. 16–17, Figure 7 · p. 17

B.4 PROMPT AND INFERENCE DETAILS

  • Inference uses stage-specific prompts and state projections; controller stages return fixed-format JSON, while generation and editing produce RGB masks.Generation and editing each separate controller instruction synthesis from execution by the frozen UMM.Source: §B.4 · p. 17
  • Before generation, the controller assesses whether anatomical knowledge is sufficient; retrieved image–mask supports provide semantic, anatomical, and appearance cues rather than templates to copy.The controller can include a retrieved support pair in the synthesized generation instruction.Source: §B.4 · pp. 17–18, Figure 9 · p. 19
  • For localized correction, the controller uses the retained mask, residual-error diagnosis, and editing history to specify a targeted revision; the editor receives the query, mask, instruction, and output contract.The editor does not directly receive the interaction trajectory or retrieved supports, and the instruction can identify reliable regions to preserve.Source: §B.4 · pp. 18–19, Figure 10 · p. 19, Figure 12 · p. 20
  • The critic compares two predictions label by label and diagnoses errors; the controller then chooses ACCEPT, EDIT, or GENERATE based on reliability and correction prospects.When EDIT is selected, the controller identifies the residual error to target.Source: §B.4 · pp. 19–20

B.5 ADDITIONAL ANALYSES FOR OBSERVATIONS

Additional analyses examine support diversity and the complementary roles of editing and resampling. MMR reduces support redundancy with a small relevance trade-off, while combining editing and regeneration yields the strongest reported refinement.

B.5 ADDITIONAL ANALYSES FOR OBSERVATIONS

  • Editing targets local errors in a promising prediction, whereas resampling explores alternatives; their combination is reported as the strongest overall refinement.The figure reports mean Dice improvement and the proportion of improved cases for regeneration, targeted editing, and their combination.Source: §B.5 · pp. 21–22, Figure 15 · p. 22
  • MMR reduces average support redundancy by 0.027, sacrifices 0.009 in query–support similarity, and raises average Top-4 mDice from 60.04% to 62.26%.The comparison finds similarity-only retrieval slightly more relevant but more redundant; MMR selects supports that remain highly relevant while being less redundant.Source: §B.5 · p. 21, Figure 15 · p. 22
  • Hypothesis dispersion is positively correlated with Best-of-4 improvement (ρ = 0.67), and broader support variation broadens the accessible hypothesis space.The figure relates query–support similarity and redundancy to prediction dispersion and its correlation with Best-of-4 improvement.Source: §B.5 · p. 21, Figure 15 · p. 22
  • Combining editing with regeneration yields a 22.2-point average improvement and improves 74.4% of cases, exceeding either method alone.Generation alone improves 16.5 mDice points and 51.3% of cases; targeted editing improves 14.8 points and 61.5% of cases.Source: Figure 15 · p. 22, §B.5 · p. 22

B.6 AGENT BEHAVIOR AND INFERENCE EFFICIENCY

SegBanana’s inference cost and controller behavior vary by dataset rather than following a fixed trajectory. Qualitative comparisons show it handling targets and modalities ranging from compact lesions to fine or multi-structure anatomy.

B.6 AGENT BEHAVIOR AND INFERENCE EFFICIENCY

  • Across datasets, SegBanana averages 1.94 inference iterations, 3.75 valid Nano Banana calls, and 2.32 retrieval calls per sample.The maximum is three iterations, but computation is dynamically determined by the evolving prediction state.Source: §B.6 · p. 22
  • 59.9% of cases terminate before the three-iteration maximum, with 46.0% stopping after the first iteration and 14.0% after the second.Task behavior varies: Stop@3 reaches 68.0% on Drishti-GS and 61.7% on ACDC, while Stop@1 reaches 66.0% on BUS-UCLM and 64.0% on Kvasir.Source: §B.6 · p. 23
  • Generation accounts for 52.5%–76.6% of controller decisions, while editing ranges from 14.4% on ACDC to 32.8% on TNBC.Editing is used more selectively to locally refine predictions that already provide a reliable basis for correction.Source: §B.6 · p. 23
  • 83.2% of cases invoke retrieval at least once, but retrieval is used for all cases on ACDC, Drishti-GS, RAVIR, and TNBC, versus 44.0% on Kvasir.ISIC requires external support in 62.0% of cases.Source: §B.6 · p. 23

B.7 ADDITIONAL CASE STUDIES

  • Qualitative examples show SegBanana generally preserving the extent and shape of compact lesions, polyps, skin lesions, and brain tumors, while several baselines under- or over-segment or confuse foreground and background.The comparisons span eight datasets and diverse targets and imaging modalities.Source: §B.7 · p. 23, Figure 16 · p. 24, Figure 17 · p. 25
  • On Drishti-GS, SegBanana better preserves the nested disc–cup relationship; on RAVIR, it better preserves thin, branching vessels than competing methods that coarsen or miss structures.Source: §B.7 · pp. 23–24, Figure 17 · p. 25
  • On TNBC, SegBanana captures a denser distribution of nuclei than methods producing sparse or nearly empty masks; on ACDC, it better preserves the relative arrangement of cardiac structures.Some ACDC baselines confuse labels or substantially distort structure extent.Source: §B.7 · pp. 24–25, Figure 16 · p. 24
Loading 2609.34235v1…