PaperlayerView exact revision on arXiv

Source-linked AI summary

Instruction-Guided Lesion Segmentation for Chest X-rays with Automatically Generated Large-Scale Dataset

Geon Choi, Hangyul Yoon, Hyunju Shin, Hyunki Park, Sang Hoon Seo, Eunho Yang, Edward Choi

arXiv:2511.15186v2cs.CV

TL;DR

CXR lesion segmentation is constrained by limited target labels and reliance on complex expert prompts. The paper introduces ILS, builds MIMIC-ILS from paired images and reports, and fine-tunes ROSALIA on the dataset. ROSALIA produces lesion masks and textual responses across diverse instructions, supporting MIMIC-ILS as a resource for CXR lesion grounding.

  • Problem

    CXR lesion-segmentation models have limited target-label coverage, while existing text-prompt approaches can require long, expert-level descriptions.

    Source: §1 · p. 1

  • Method

    The authors use an automated multimodal pipeline on paired CXR images and reports to build MIMIC-ILS, then fine-tune LISA as ROSALIA for instruction-guided lesion segmentation.

    Source: §1 · p. 2

  • Results

    ROSALIA generates accurate lesion segmentations and textual responses across diverse user instructions.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    MIMIC-ILS and ROSALIA offer a foundation for research on fine-grained lesion grounding in the CXR domain.

    Source: §7 · p. 8

Abstract

from arXiv · show

The applicability of current lesion segmentation models for chest X-rays (CXRs) has been limited both by a small number of target labels and the reliance on complex, expert-level text inputs, creating a barrier to practical use. To address these limitations, we introduce instruction-guided lesion segmentation (ILS), a medical-domain adaptation of referring image segmentation (RIS) designed to segment diverse lesion types based on simple, user-friendly instructions. Under this task, we construct MIMIC-ILS, the first large-scale instruction-answer dataset for CXR lesion segmentation, using our fully automated multimodal pipeline that generates annotations from CXR images and their corresponding reports. MIMIC-ILS contains 1.1M instruction-answer pairs derived from 192K images and 91K unique segmentation masks, covering seven major lesion types. To empirically demonstrate its utility, we present ROSALIA, a LISA model fine-tuned on the MIMIC-ILS dataset. ROSALIA can segment diverse lesions and provide textual explanations in response to user instructions. The model achieves high accuracy in our newly proposed task, highlighting the effectiveness of our pipeline and the value of MIMIC-ILS as a foundational resource for pixel-level CXR lesion grounding. The dataset and model are available at https://github.com/checkoneee/ROSALIA.

1. Introduction

Existing CXR lesion-segmentation approaches are limited in lesion coverage and often require expert-level prompts, motivating instruction-guided lesion segmentation (ILS). The paper introduces an automated dataset pipeline and ROSALIA, a model intended to handle diverse user instructions with lesion masks and textual outputs.

1. Introduction

  • Prior CXR text-prompt methods target a single lesion type and require long, expert-level descriptions, limiting their practical use for diverse users.The paper notes that such inputs can be difficult for non-experts who cannot interpret CXRs themselves.Source: §1 · pp. 1–2
  • ILS asks models to follow varied instructions about lesion type, location, or global abnormalities, report absent lesions, and provide textual descriptions.A dataset for this task was unavailable, in part because masks must be paired with precise lesion and location instructions.Source: §1 · p. 2
  • MIMIC-ILS provides 1.1M instruction–answer pairs from 192K images and 91K lesion masks, with expert evaluations reporting over 95% acceptance.It pairs each instruction–answer set with an auto-labeled mask and detailed lesion profile.Source: §1 · p. 2
  • The authors construct MIMIC-ILS with a fully automated pipeline that uses paired CXR images and reports to generate lesion masks and instruction–answer pairs.The pipeline combines pretrained vision models and large language models to extract anomalous regions and structured text.Source: §1 · p. 2
  • Fine-tuned on MIMIC-ILS, ROSALIA generates lesion masks and textual descriptions for specific, generic, and absence-confirmation requests.The paper presents it as the first vision–language model designed for ILS in CXRs.Source: §1 · p. 2

2. Related Work

The related work contrasts scarce, narrow CXR lesion-mask datasets with broader text-directed segmentation methods. Medical-domain approaches remain constrained in their prompt capabilities and often assume users have already interpreted the image.

2.1. Lesion Segmentation and Datasets

  • CXR pixel-level lesion annotations are scarce: available datasets often provide only bounding boxes or masks for a single lesion type.The paper links this limited coverage to the restricted effective segmentation range of models trained on existing datasets.Source: §2.1 · p. 3

2.2. Referring Image Segmentation

  • Referring image segmentation (RIS) segments a target specified by text, and newer vision–language models can handle instructions requiring real-world knowledge.Earlier approaches aligned image features with text labels to generate masks.Source: §2.2 · p. 3
  • Medical RIS approaches remain limited: CXR systems use free-form lesion descriptions that assume users have already reviewed the image, while other medical prompts cannot handle sentence-level instructions.The paper contrasts these approaches with its model, which aims to provide lesion mask, presence, and type information without prior image interpretation.Source: §2.2 · p. 3

3. Automatic Dataset Construction

The dataset pipeline extracts report and image information to generate verified lesion masks, then uses those masks to create valid instruction–answer pairs for seven CXR lesion types.

3. Automatic Dataset Construction

  • The fully automated construction process first extracts textual and spatial information from paired CXR images and reports, generates lesion masks, verifies them, and then builds instruction–answer pairs.The framework is designed to operate without explicit image annotations or human intervention.Source: §3 · p. 3, §3.1 · p. 3

3.1. Grounded Lesion Mask Generation

  • Report sentences are converted into six-element tuples with mapped anatomical locations, while RadEdit, CXAS, and YOLO provide anomaly maps, anatomy masks, and lesion boxes.These textual and spatial cues supply the inputs for lesion-mask generation.Source: §3.1 · p. 3
  • Candidate boxes are filtered by anatomy overlap, confidence, anomaly-signal ratio, and lung-relative size; intersecting anomaly components are then refined into lesion masks.Anomaly maps can miss parts of actual lesions, so the signal-ratio and size conditions help avoid excessively small or empty masks.Source: §3.1 · p. 4
  • Verification labels reported locations as grounded when they overlap a generated mask, while unreported lung regions become empty locations for negative samples.Reported-but-ungrounded locations are distinguished from both grounded and empty locations.Source: §3.1 · pp. 4–5

3.2. Instruction-Answer Pair Generation

  • MIMIC-ILS pairs masks with positive and negative instructions for seven lesion types, including cardiomegaly, pneumonia, atelectasis, opacity, consolidation, edema, and effusion.Positive pairs include a ground-truth lesion mask; negative pairs use an empty mask.Source: §3.2 · p. 5
  • The framework generates basic, global, and lesion-inference instructions only when the available grounding supports them, and supplies mask and textual-description answers.Global instructions require the grounded and reported locations to match; negative examples can use absent or explicitly negated findings or empty locations.Source: §3.2 · pp. 5–6

4. MIMIC-ILS Dataset

MIMIC-ILS contains 1.1M instruction-answer pairs from 192K MIMIC-CXR images, with 91K unique masks spanning seven lesion types. Expert review found 96.4% of 10.7K reviewed mask samples acceptable for the test set.

Dataset composition

  • 1.1M instruction-answer pairs (135K positive and 930K negative) come from 192K images and 91K unique lesion masks across seven lesion types.The dataset is split into 1M training, 8.2K validation, and 12K test samples.Source: §4 · p. 6
  • Figure 4 plots sample counts by lesion type for cardiomegaly, pneumonia, atelectasis, opacity, consolidation, edema, and effusion.Each mask can be associated with multiple instruction-answer samples.Source: Figure 4 · p. 6, §4 · p. 6

Human evaluation

  • 96.4% of 10.7K initially reviewed mask samples were rated acceptable and included in the final test set.Four radiation oncologists reviewed the samples; any sample judged unacceptable by at least one expert was excluded.Source: §4 · p. 6

5. Model Training

ROSALIA adapts LISA by combining a vision-language model with SAM to produce text and lesion masks from CXR images and instructions. Its training objective combines answer-text and segmentation losses.

Architecture

  • ROSALIA adopts LISA’s architecture, integrating a vision-language model (VLM) backbone with SAM for instruction-guided mask prediction.The model is trained using LISA’s architecture, with the VLM processing the image and instruction.Source: §5 · pp. 6–7, Figure 5 · p. 7
  • The VLM produces a textual description and a [SEG] token from the CXR and instruction, then passes the token embedding to SAM to generate the mask.SAM’s frozen image encoder extracts image embeddings, which its mask decoder combines with the [SEG] embedding.Source: §5 · p. 7, Figure 5 · p. 7

Training objective

  • Training combines autoregressive cross-entropy for answer text with segmentation loss comprising binary cross-entropy and DICE losses.The overall objective weights the text loss by λtxt and adds the mask loss.Source: §5 · p. 7

6. Experiments

On MIMIC-ILS, ROSALIA outperforms general- and medical-domain baselines on segmentation, while also showing strong lesion-type and text-response performance. Qualitative examples show selective segmentation across instructions and correct handling of empty targets.

6.1. Implementation Details

  • The evaluation uses gIoU and cIoU for positive samples and empty-target accuracy for negative cases, against general- and medical-domain segmentation baselines.Training uses LISA-7B with LLaVA and SAM-H, LoRA fine-tuning for the VLM, and a fully fine-tuned mask decoder; positive and negative samples are balanced per mini-batch.Source: §6.1 · p. 7

6.2. Main Results

  • ROSALIA achieves notably high MIMIC-ILS test performance, while general- and medical-domain baselines have low positive-case IoU and near-zero empty-target accuracy in most instances.The comparison is reported on the MIMIC-ILS test set; ROSALIA is LISA-7B fine-tuned on MIMIC-ILS.Source: Table 4 · p. 7, §6.2 · p. 7

6.3. Qualitative Results

  • Figure 6 shows baselines often producing incorrect masks or segmenting whole lung regions, while ROSALIA targets the instructed lesion and identifies empty-target cases.The first three rows are positive cases and the last row is a negative case with an empty target mask.Source: Figure 6 · p. 8, §6.3 · p. 8

6.2. Main Results

  • Overall gIoU exceeds 0.7 across lesion types, and even the lowest lesion-type gIoU exceeds 0.55.The passage interprets these scores as more than 80% and 70% regional overlap, respectively, when predicted and ground-truth masks are similar in size.Source: §6.2 · p. 7, Table 5 · p. 8
  • ROSALIA achieves high text-response accuracy across most question types under an exact-match criterion for both templates and all variables.The reported evaluation compares responses with structured ground-truth information.Source: Table 6 · p. 8, §6.2 · p. 8

6.3. Qualitative Results

  • On the same image with multiple lesions, ROSALIA selectively segments the lesion and location requested by each instruction.Figure 7 presents outputs from different instructions applied to the same image.Source: Figure 7 · p. 8, §6.3 · p. 8

7. Conclusion

The conclusion presents MIMIC-ILS as a large-scale instruction-guided CXR lesion segmentation dataset and ROSALIA as a model trained for the task. It reports high expert acceptance and accurate segmentations and textual responses across instructions.

7. Conclusion

  • MIMIC-ILS and ROSALIA are presented as a foundation for CXR lesion grounding, with expert evaluations reporting high acceptance and ROSALIA generating accurate segmentations and textual responses.The dataset is described as constructed through a fully human-free process.Source: §7 · p. 8

A.1. Report Pre-Processing

  • Reports are preprocessed to extract findings, impression, and last-paragraph sections, then a fallback hierarchy selects one representative section or excludes the study.Findings is preferred, followed by impression and then the last paragraph.Source: §A.1 · p. 13

A.2. Large Language Models and Prompts

  • Two LLMs structure report findings as six-element tuples and normalize lesion locations in a two-step process.Mistral-Small-3.1-24B-Instruct-2503 handles structuring; medical-domain medgemma-27b-text-it handles location mapping, both run locally under the data-use agreement.Source: §A.2 · p. 13

A.3. Vision Models and Characteristics

  • HybridGNet supplies organ masks, RadEdit creates opacity-removed images, CXAS segments anatomy on edited images, and YOLO detections are filtered to selected hyperintense lesions.RadEdit used original MIMIC-CXR rather than histogram-equalized JPG images because the latter significantly degraded edited-image quality.Source: §A.3 · p. 13

A.4. Thresholds for Lesion Mask Generation

  • Physician quality checks calibrated mask-generation thresholds iteratively, with lower thresholds for edema because it tends to spread widely through the lungs.The passage says finalized thresholds are summarized in Table 7 but does not provide their values.Source: §A.4 · p. 13, Table 7 · p. 13

A.5. Lesion Mask Post-Processing

  • Post-processing removes scattered noise with erosion and dilation, expands masks to similar-intensity adjacent pixels, and extends lung-base effusion coverage to the costophrenic angle.The illustrated example discards a lesion box with insufficient overlap against the reported upper-lung location before filtering the anomaly map.Source: §A.5 · pp. 13–14, Figure 8 · p. 14

A.6. Empty Location

  • An “empty location” is a lung region with no overlap between the reported lesion location and its corresponding anatomy mask; this information is computed for all reported lesions.The empty-location information is used in a subsequent data-generation step.Source: §A.6 · p. 14

A.7. Sample Discarding and Pipeline Recall

The pipeline favors high-confidence samples over recall, then uses verified image and mask inputs to build lesion-type and location-aware instructions. The section describes positive and negative prompts and their intended clinical utility.

A.7. Sample Discarding and Pipeline Recall

  • A higher-recall training dataset degraded MIMIC-ILS test performance (gIoU: 54.3%, cIoU: 61.4%, N-Acc: 95.8%), so the pipeline uses a high threshold to favor high-confidence samples.The authors state recall could be increased by relaxing cross-model consistency criteria, but chose not to maximize it.Source: §A.7 · p. 14, Table 8 · p. 14

B. Lesion Types

  • The dataset covers seven major lesions: opacity, consolidation, pneumonia, atelectasis, edema, cardiomegaly, and effusion.The categories were identified through discussions with board-certified physicians.Source: B. Lesion Types · p. 14

C.1. Positive Instruction

  • Positive prompts pair grounded lesions with locations, substitute opacity for tentative findings, and allow global prompts when generated and reported locations match, always for cardiomegaly.Lesion-inference prompts target pneumonia, atelectasis, and edema because reports often describe these findings inferentially; certainty appears in the ground-truth text.Source: §C.1 · p. 17

C.2. Negative Instruction

  • Negative prompts cover lesions absent or negated in reports and pair target lesions with empty locations, with at most one negative instruction per lesion type per study..Source: §C.2 · p. 17

C.3. Clinical Utility of MIMIC-ILS

  • On the paraphrased test set, LISA achieves gIoU: 67.3%, cIoU: 73.1%, and N-Acc: 96.5%; global and negative prompts support screening without visual inspection.The authors propose iterative queries across anatomical locations to check lesion presence or absence.Source: §C.3 · p. 17

D.1. Chest X-rays

  • The pipeline uses PA and AP MIMIC-CXR images and excludes noisy or truncated cases using CXReasonBench metadata on anatomy masks and chest visibility..Source: §D.1 · p. 18

D.2. Lung and Heart Masks

  • Cases with substantial discrepancies between HybridGNet and CXAS organ masks are excluded because these errors could disrupt grounded lesion-mask generation.The checks include outermost lung-mask x-coordinates and lowermost heart-mask y-coordinates.Source: §D.2 · p. 18

D.3. Cardiomegaly

  • Negative cardiomegaly samples are retained only when the measured cardiothoracic ratio is 0.45 or less, using a physician-calibrated cutoff.The ratio is measured from the right lung, left lung, and heart masks produced by HybridGNet.Source: §D.3 · p. 18

E. MIMIC-ILS Dataset

The dataset section identifies the split and instruction-type distributions and describes a physician review of test samples. Four physicians with lesion-contouring experience conducted the evaluation.

E.1. Details for Dataset Splits

  • Tables 10–12 report MIMIC-ILS instruction-answer pair counts by lesion and template type for train, validation, and test splits.Test counts reflect exclusions made during quality assessment.Source: §E.1 · p. 18

E.2. Quality Assessment

  • Four physicians reviewed all positive test samples and divided negative samples among themselves, judging each against the image, lesion, location, and report independently.Reviewers marked each sample pair “Acceptable” or “Not Acceptable.”Source: §E.2 · p. 18

E.3. Details on Expert Evaluation

  • The evaluators were radiation oncologists experienced in lesion contouring, and Table 13 reports lesion-level acceptance rates by expert.Two were board-certified, with 9 and 7 years of experience; two were residents, each with 6 years.Source: §E.3 · p. 18

E.4. Data Generation Examples

The data-generation examples include varied lesion types and anatomical locations, positive and negative cases, and structured information.

E.4. Data Generation Examples

  • Figure 11 shows representative generated samples across lesion types and locations, including positive and negative cases with corresponding structured information..Source: §E.4 · p. 18, Figure 11 · p. 20
  • Tables 10–12 enumerate generated instruction-answer pairs by lesion and template type across the train, validation, and test splits..Source: Table 10 · p. 19, Table 11 · p. 19, Table 12 · p. 19
  • Table 13 presents human-evaluation acceptance rates by lesion type for each expert..Source: Table 13 · p. 19

F. Model Training Details

Training used two NVIDIA H100 GPUs for 15 epochs, while inference on 12K test examples took about 20 minutes for segmentation alone or 1.5 hours with text outputs.

F. Model Training Details

  • Two and a half days on two NVIDIA H100 GPUs were required to train the LISA-7B–based model for 15 epochs.Training used DeepSpeed Stage-2 with WarmupDecayLR, 100 warmup steps, and learning rates from 0 to 0.0003.Source: F. Model Training Details · p. 21
  • About 20 minutes were needed for segmentation-only inference on the 12K-example test set, versus approximately 1.5 hours with text outputs.The longer time applies to inference that generates both segmentation and text.Source: F. Model Training Details · p. 21
  • Each training image had a 50% chance of being processed with histogram equalization.Source: F. Model Training Details · p. 21

G. Additional Experimental Results

ROSALIA achieves consistently high text-response accuracy across most lesion and question types, while qualitative examples show instruction-tailored segmentation and factual text outputs.

G.1. Lesion-Wise Text Accuracy

  • Across most lesion and question types, ROSALIA consistently achieves high text-response accuracy, similar to the segmentation performance reported in Table 5.The discussion notes that lesion-inference accuracy is naturally limited because CXR alone often cannot provide a definitive diagnosis.Source: §G.1 · p. 21, Table 14 · p. 21

G.2. Additional Qualitative Examples

  • ROSALIA produces accurate segmentation tailored to diverse instructions, unlike the baseline models in the qualitative comparison.Figure 12 presents the qualitative comparison of ROSALIA and baseline segmentation results.Source: §G.2 · p. 21, Figure 12 · p. 22
  • Examples show factual text responses alongside precise segmentation, and the generated responses match ground-truth answers in expert-rated examples.Figure 13 presents examples of ROSALIA's textual responses.Source: §G.2 · p. 21, Figure 13 · p. 23

Related papers

Loading 2511.15186v2…