PaperlayerView exact revision on arXiv

Source-linked AI summary

XFlow: A Workflow Model for Instruction-Guided Lesion Segmentation in Chest X-rays

Geon Choi, Hangyul Yoon, Hyunki Park, Sang Hoon Seo, Edward Choi

arXiv:2609.34513v1cs.CV

TL;DR

CXR lesion segmentation requires pixel-level judgments, while existing medical text-guided models have limited coverage and ROSALIA can produce noisy single-shot masks. XFlow follows a coarse-to-fine workflow, combining box localization with point refinement; it achieves the best segmentation quality on internal and external evaluation, including against ROSALIA trained on the same annotations.

  • Problem

    Medical text-guided segmentation covers a limited range of CXR targets, and ROSALIA’s single-shot masks remain limited in quality and can contain scattered noise.

    Source: §1 · pp. 1–2

  • Method

    XFlow detects lungs and checks for the queried finding, prompts fine-tuned SAM with a lesion box for an initial mask, then refines it through point prompts with intermediate decisions visible.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    XFlow outperforms compared models on internal and external evaluation; on CheXpercept, its gIoU rises from 0.621 to 0.703 against ROSALIA.

    Source: §1 · p. 2, §5.3 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    The intermediate steps and masks XFlow produces could serve as a foundation for other CXR tasks.

    Source: §7 · p. 9

    The IBISAgent comparison uses an intermediate released checkpoint that cannot be reliably retrained.

    Source: §5.4 · p. 7

Abstract

from arXiv · show

Existing text-guided segmentation models in the medical domain cover only a narrow set of anatomical structures and lesions in chest X-rays (CXRs), and most of them assume that the queried target is always present in the image. Instruction-guided lesion segmentation (ILS) was introduced to overcome these limitations by segmenting diverse lesion types from simple user instructions while also recognizing when the queried lesion is absent, and ROSALIA was proposed as the first model for this task. However, the masks produced by ROSALIA remain of limited quality, often carrying scattered noise. Moreover, ROSALIA predicts the mask in a single shot, which differs fundamentally from how radiologists perceive and delineate lesions in practice. A radiologist first surveys the entire thorax, then localizes the approximate region of abnormality, and only then refines the lesion contour. Motivated by this coarse-to-fine, multi-level perception process, we present XFlow, a workflow model for ILS that combines box-based localization with multi-turn point refinement. XFlow detects the lungs, decides whether the queried finding is present in each of them, and prompts a fine-tuned SAM with the lesion box for an initial mask. It then corrects that mask through point prompts until its boundary follows the lesion, leaving every intermediate decision visible. Our experiments show that XFlow achieves the best segmentation quality on both internal and external evaluation. Notably, it surpasses ROSALIA in segmentation quality even when the two are trained on the same lesion annotations. Code and model weights will be made publicly available.

1. Introduction

Medical CXR lesion segmentation is demanding, while existing text-guided models cover limited targets, often assume the finding is present, and may produce noisy single-shot masks. XFlow uses a radiologist-inspired, two-stage workflow and outperforms compared models on internal and external evaluation, including ROSALIA trained on the same annotations.

1. Introduction

  • Medical CXR lesion segmentation demands pixel-level judgment in overlapping, low-resolution images and consumes the largest share of radiologists’ reading time.Radiologists must also interpret many studies under time pressure.Source: §1 · p. 1
  • Existing medical text-guided models cover limited CXR targets and often assume the queried finding is present, while ROSALIA’s masks can contain scattered noise.ILS addresses the presence question, but ROSALIA’s mask quality remains limited.Source: §1 · p. 1
  • Radiologists progress from surveying the thorax to localizing an approximate abnormality and then refining its boundary, unlike single-shot or point-only approaches.The authors position XFlow’s coarse-to-fine workflow as following these successive levels of perception.Source: §1 · pp. 1–2
  • XFlow first detects lungs, checks each for the queried finding, and boxes the lesion for an initial SAM mask, then refines it through point prompts while exposing intermediate decisions.The sequence separates coarse localization from fine boundary correction.Source: §1 · p. 2, Figure 1 · p. 2
  • XFlow outperforms all compared models in segmentation quality on internal and external evaluation, with each workflow stage and its training scheme contributing to the gain.It also surpasses ROSALIA when both use the same MIMIC-ILS cases, so the improvement does not come from additional lesion annotations.Source: §1 · pp. 2–3

2. Related Works

Prior CXR lesion segmentation methods either predict masks in one step or refine them with points alone. XFlow combines box localization with multi-turn point refinement and makes its intermediate decisions inspectable.

2. Related Works

  • General-domain methods use either single-step geometric prompts or multi-turn point clicks, and medical IBISAgent follows the point-only approach.A single-step prompt cannot correct its resulting mask, while point-only refinement carves the mask click by click.Source: §2 · p. 3
  • XFlow combines lesion localization with a box and successive point-based boundary refinement within at most four steps, using a reward defined over the full trajectory.IBISAgent averages 8.69 steps and optimizes turns in isolation; its released checkpoint is intermediate and cannot be reliably retrained.Source: §2 · p. 3
  • Earlier CXR text-guided lesion models were limited to a single target, while ROSALIA covers seven findings but often produces imprecise one-shot masks.The findings include cardiomegaly, pneumonia, atelectasis, opacity, consolidation, edema, and effusion.Source: §2 · p. 3

3. Task Formulation

Instruction-guided lesion segmentation maps an image and instruction to a binary mask for the specified region. Unlike referring-image segmentation, the queried finding may be absent, in which case the ground-truth mask is empty.

3. Task Formulation

  • Given a chest X-ray and an instruction specifying a lesion and where to look, the model returns a binary mask over the referred region.The location can range from the whole thorax to one lung or a specific zone.Source: §3 · p. 3
  • ILS differs from referring-image segmentation because the queried finding may be absent, making the ground-truth mask empty.The model must decide whether the finding exists in the specified region rather than assume it is present.Source: §3 · p. 3

4. XFlow

XFlow separates lesion segmentation into sequential initial-localization and mask-revision policies that interact with a shared promptable segmenter through boxes and points. Its training combines a fine-tuned segmenter with separately trained policies, each using supervised fine-tuning followed by reinforcement learning.

4.1. Architecture Overview

  • The initial policy grounds the relevant lung regions, judges lesion presence, and localizes present lesions with boxes that prompt the shared segmenter.For absent lesions it emits no lesion box, corresponding to an empty mask; thorax-wide queries ground both lungs, while side-specific queries ground one.Source: §4.1 · p. 4
  • The revision policy uses up to four steps of positive or negative point prompting, conditions on the preceding interaction history, and unions revised masks into the final prediction.Positive points expand the predicted region; negative points shrink it, and the loop can stop when the policy judges the mask sufficient.Source: §4.1 · p. 4

4.2. Training Data

  • MIMIC-ILS data for SFT and RL are constructed automatically from existing masks and metadata, with disjoint training and evaluation splits.The RL and evaluation sets are drawn from cases identified by medical experts as having the most reliable masks.Source: §4.2 · p. 4

4.3. SAM Training

  • SAM3 is fine-tuned on the SFT split using sampled enclosing boxes and inside/outside points to match policy prompts at inference.The segmenter is shared by both policies, which interact with it through geometric prompts rather than predicting masks directly.Source: §4.1 · p. 4, §4.3 · p. 5

4.4. VLM Training

  • Both policies are trained separately with supervised fine-tuning followed by reinforcement learning, while the segmenter remains frozen.The authors use RL because SFT alone teaches target-sequence imitation without direct feedback on decision correctness.Source: §4.4 · p. 5, Figure 2 · p. 5

4.4.1. Initial Segmentation Policy

  • The initial-policy reward combines box IoU with output-format and duplicate-box terms, while negative cases receive a binary box reward based on emitting no lesion box.Lung boxes come from an off-the-shelf model; presence labels and lesion boxes derive from MIMIC-ILS.Source: §4.4.1 · pp. 5–6

4.4.2. Segmentation Revision Policy

  • Revision-policy examples simulate imperfect starting masks and iterative user clicks, while its reward measures trajectory-level IoU gain plus format compliance.Rewarding net improvement lets the policy stop when further revision would no longer help.Source: §4.4.2 · p. 6

5. Experiments

XFlow achieves the highest segmentation quality on both evaluation benchmarks, including against ROSALIA trained on the same cases. Its gains are supported by stage-wise ablations, while qualitative examples show more faithful lesion contours than the baselines.

5.1. Implementation Details

  • Training fine-tunes SAM3 with its image encoder unfrozen and trains both policies on MedGemma-1.5-4B-it; revision-policy rollouts use the segmenter for at most four clicks.The revision policy is warm-started from the initial policy’s supervised-fine-tuning checkpoint.Source: §5.1 · p. 6

5.2. Experimental Setup

  • Evaluation uses unseen, physician-vetted CheXpercept cases at chest, lung, and zone specificity, plus external CheXlocalize chest-level cases; comparisons include general- and medical-domain models.CheXpercept’s three question levels are equally weighted; the human benchmark has no F1 because radiologists were told which finding to draw. IBISAgent is an intermediate checkpoint whose training pipeline is not fully released and whose data construction uses physician annotations.Source: §5.2 · pp. 6–7, Table 1 · p. 7, §5.4 · p. 7

5.3. Main Results

  • XFlow has the highest gIoU and cIoU on both benchmarks; versus same-data ROSALIA, gIoU rises from 0.621 to 0.703 on CheXpercept and 0.233 to 0.283 on CheXlocalize.On CheXlocalize, cIoU rises from 0.289 to 0.331, approaching the human benchmark’s 0.339; F1 remains on par at 0.632 versus 0.634.Source: §5.3 · p. 7, Table 1 · p. 7
  • Most of the gain comes from Stage 1; Stage 2 adds 0.010 gIoU on CheXpercept and 0.003 on CheXlocalize.Stage 2 only makes local corrections, cannot recover a lesion missed by Stage 1, and inherits Stage 1’s F1 because it does not change presence decisions.Source: §5.3 · p. 7, Table 2 · p. 8

5.4. Qualitative Results

  • XFlow recovers small lesions near the heart and follows ground-truth boundaries with fewer stray regions, while ROSALIA misses small lesions and produces imprecise, fragmented masks.Other baselines largely fail to follow the instruction or delineate the requested lesion.Source: §5.4 · pp. 7–8, Figure 3 · p. 8

6. Further Analyses

XFlow reduces mask fragmentation relative to ROSALIA and receives expert rankings close to ground truth, with a stronger preference than for ROSALIA.

6.1. Mask Noise

  • ROSALIA produces 5.63 excess components and 5.57 specks per case, compared with XFlow’s 1.71 excess components and 1.69 specks.Specks account for 2.6% of ROSALIA’s mask area and 0.14% of XFlow’s; 3.43 versus 1.06 specks per case touch no ground-truth area.Source: §6.1 · pp. 8–9, Table 3 · p. 9

6.2. Expert Preference Study

  • Three physicians ranked blinded ground-truth, XFlow, and ROSALIA masks for the same 95 cases, with ties allowed.The cases represented each finding and question level equally.Source: §6.2 · p. 9, Table 4 · p. 9
  • XFlow is preferred over ROSALIA 2.01 times as often and sits close to ground truth, with a 0.141 Davidson difference.The table reports that readers who separate XFlow from ground truth prefer ground truth 1.15 times as often.Source: §6.2 · p. 9, Table 4 · p. 9

7. Conclusion

XFlow separates coarse lesion localization from fine delineation in a radiologist-like sequence, outperforming prior models, including ROSALIA with matched lesion annotations.

7. Conclusion

  • XFlow separates coarse localization from fine-grained delineation and performs these stages sequentially rather than mapping an image to a mask in one step.The workflow mirrors the way a radiologist reads an image.Source: §7 · p. 9
  • XFlow substantially outperforms prior models and surpasses ROSALIA even when both are trained on the same lesion annotations.Source: §7 · p. 9
  • The intermediate steps and masks produced by XFlow could serve as a foundation for other CXR tasks.Source: §7 · p. 9

A.1. Data Curation

The data curation process separates supervised training from reinforcement-learning and evaluation data, then filters the latter using agreement among three sources.

A.1. Data Curation

  • All 2,100 CheXpercept radiographs are held out of the supervised split, with the curated cases divided between reinforcement learning and evaluation.The 1,856 radiographs remaining after filtering are split by patient into 1,296 reinforcement-learning and 560 evaluation radiographs, with no patient in both.Source: §A.1 · p. 13, Table 5 · p. 13
  • Removing CheXpercept from MIMIC-ILS leaves 185,488 radiographs for supervised training and keeps its 1,508 validation radiographs untouched.Source: §A.1 · p. 13

A.2. Three-Source Agreement

  • A radiograph-finding-region label is retained only when CheXpercept, a parsed radiology report, and Chest ImaGenome agree on presence or absence.Any disagreement or missing source is dropped; 1,856 of 2,100 radiographs retain at least one admitted triple.Source: §A.2 · pp. 13–14

A.3. Building the Training Examples

Training examples are constructed from existing radiographs, masks, and labels, with separate examples for initial box-based segmentation and multi-turn mask revision.

A.3. Building the Training Examples

  • Examples for each stage are automatically generated from the radiograph, lesion mask, and report-derived label already carried by each split.A revision example is a whole trajectory of up to four turns, whereas an initial-segmentation example is one instruction.Source: §A.3 · p. 14, Table 6 · p. 14
  • Initial-segmentation examples pair an X-ray and generated question with target lung and lesion boxes derived from lung masks and the annotated finding mask.The question’s location scope determines which lung boxes are included.Source: §A.3 · p. 14
  • Revision examples alternate point prompts with updated mask observations, forming a conversation that begins with the initial mask and the same user question.The initial mask is synthesized by perturbing the ground-truth lesion box and passing it with the image through the segmenter.Source: §A.3 · p. 14
  • The simulated oracle clicks the deepest point in whichever error region is larger, using positive clicks for missed regions and negative clicks for over-segmented regions.It iteratively prompts the segmenter to produce an improved mask and forms the revision trajectory from alternating masks and clicks.Source: §A.3 · p. 14
  • Each simulated trajectory is limited to four clicks and stops when IoU reaches 0.70; repeated same-polarity clicks within 25 pixels and non-improving trajectories are excluded.Source: §A.3 · pp. 14–16

A.4. Training Details

XFlow trains a segmenter and two policies, with different optimization choices for segmentation, supervised fine-tuning, and reinforcement learning.

A.4. Training Details

  • Table 7 lists hyperparameters for XFlow’s segmenter F_seg and its initial-segmentation and revision policies, each trained in two phases.The RL phase starts from the same policy’s SFT checkpoint; effective batch size accounts for per-device batch, gradient accumulation, and eight GPUs.Source: Table 7 · p. 16, §A.4 · p. 16
  • F_seg trains at 1024 × 1024 with the SAM 2 loss and selects the lowest-loss of three mask hypotheses per prompt.The loss is 20 · ℒfocal + ℒdice + ℒIoU; the encoder remains unfrozen, with a 0.6× decoder learning rate and 0.9 layer-wise trunk decay.Source: §A.4 · p. 16
  • Both policies use LoRA in every linear layer, but SFT also updates the vision tower while RL freezes it and restricts adapters to language layers.During SFT, the vision tower uses 0.3× the base rate because the policies emit image-coordinate boxes and clicks and the tower has not seen chest radiographs.Source: §A.4 · p. 16

B.1. Evaluation Set Construction

The evaluation set derives questions from agreed annotations on held-out radiographs, while qualitative comparisons and expert rankings assess predicted masks.

B.1. Evaluation Set Construction

  • Each radiograph in the 560-image held-out set can contribute multiple questions, with findings queried at applicable location-specificity levels.Questions are formed from triples agreed on by the three sources; positive rows carry masks and are scored by IoU, while negative rows require an empty mask.Source: §B.1 · p. 17, Table 8 · p. 17
  • Zone-level questions account for 68.7% of negatives, versus 9.3% at chest level, so the three location levels are averaged equally rather than pooled.More specific questions can be negative even when a finding is present elsewhere; pooling would emphasize zone-level declines and barely register performance on questions without a specified location.Source: §B.1 · p. 17

B.2. Additional Qualitative Examples

  • Figure 6 presents additional predicted-mask comparisons across models using the layout of Figure 3.The supplied passage identifies the comparison and layout, but does not state a further outcome.Source: §B.2 · p. 17

C.1. Labeling Process

  • Readers ranked ground truth, ROSALIA, and XFlow masks shown anonymously in random order alongside the X-ray.Cases were randomly selected from positives with a ground-truth mask where both models returned a mask; each reader received a panel and annotation sheet.Source: §C.1 · p. 18

C.2. Expert Profiles

  • The three readers were board-certified radiation oncologists with 8, 4, and 4 years of clinical experience.Source: §C.2 · p. 18

C.3. Preference Modeling

Preference modeling accounts for ties in expert rankings of masks, which ordinary Bradley–Terry comparisons do not represent. Davidson and Rao–Kupper models assign ties distinct probabilities and are fit to the pairwise outcomes.

C.3. Preference Modeling

  • Bradley–Terry has no tie outcome, and scoring ties as half-wins discards the readers’ inability to distinguish masks and counts one observation as two.The authors therefore fit two models with a separate tie probability.Source: §C.3 · p. 18
  • Davidson treats a tie as a third outcome; ν governs tie frequency, and its reciprocal indexes how finely a reader discriminates.Its tie term is proportional to the geometric mean of the two mask strengths, making a tie most likely for an evenly matched pair; ν = 0 recovers Bradley–Terry.Source: §C.3 · p. 18
  • Rao–Kupper models a tie as neither mask being far enough ahead to win, with θ ≥ 1 setting the required gap.It reduces to Bradley–Terry at θ = 1, and unlike Davidson its tie probability is asymmetric in the two strengths.Source: §C.3 · p. 18
  • Both tie-aware models are fit by maximum likelihood over 855 pairwise outcomes, jointly estimating tie parameters and mask strengths.For a 9.9% observed tie rate, the estimates are ν = 0.232 and θ = 1.241; intervals use 2,000 case-level bootstrap resamples because comparisons from one ranking are not independent.Source: §C.3 · p. 18
Loading 2609.34513v1…