Source-linked AI summary
Instance-Guided Report Anchoring for Text-Free 3D Abnormality Segmentation in Chest CT
Zhenyu Bu, Haoyan Ding, Chushu Shen, Xinyuan Zheng, Peiyu Duan, Xueqi Guo, Sepehr Farhand, Yoshihisa Shinagawa, Gerardo Hermosillo, Chaowei Wu
TL;DR
Chest CT segmentation needs costly voxel-level labels, while existing report-guided methods require text at inference and typically localize one finding at a time. IGRA preserves instance–finding correspondence during training, removes text components for image-only inference, and reformulates grounding as multi-label segmentation. It improves Dice over the strongest image-only baseline, performs comparably to VoxTell on single findings, and generalizes across backbones and external datasets.
Problem
Accurate 3D abnormality segmentation requires costly dense voxel-level labels, while existing grounding methods depend on report-derived findings at inference and query one finding per pass.
Method
IGRA anchors each abnormality instance representation to its corresponding report-finding embedding during training, then removes text components and predicts merged category-level masks from CT alone.
Results
IGRA improves Dice by 22.5% over TransUNet, from 25.25 to 30.93, and consistently improves Dice and hit rate across four backbones with zero-shot gains on three additional datasets.
Takeaways & Limitations
Report information can provide instance-specific training guidance while the deployed segmentation model remains image-only and predicts all abnormality categories in one pass.
Takeaways & Limitations
The formulation is restricted to ReXGroundingCT’s 14 abnormality categories and requires paired finding sentences and lesion masks during training.
Abstract
from arXiv · showhide
Accurate 3D abnormality segmentation in chest CT requires dense spatial supervision, but obtaining expert voxel-level labels is costly. Radiology reports, however, are routinely generated during clinical interpretation and contain instance-specific descriptions that can provide additional guidance without new dense annotation. Existing vision-language grounding methods typically require report-derived findings at inference, making localization dependent on paired text and limiting each forward pass to a queried finding. We propose Instance-Guided Report Anchoring (IGRA), a model-agnostic module that preserves the correspondence between each annotated abnormality instance and the report finding that describes it. IGRA pools each instance representation and anchors it to the corresponding finding embedding during training; all text-related components are discarded at inference. We further reformulate free-text grounding on ReXGroundingCT as multi-label volumetric segmentation by merging same-category instances, allowing all abnormality categories to be predicted in one image-only forward pass. IGRA improves Dice by 22.5% over the strongest image-only baseline (30.93 vs. 25.25) and is comparable to VoxTell on the single-finding subset (30.29 vs. 30.43). Applied unchanged to four standard 3D segmentation backbones, IGRA improves Dice and hit rate across all architectures. Zero-shot evaluation on LIDC-IDRI, PleThora, and a private in-house dataset further shows consistent gains over image-only baselines.
1. Introduction
IGRA uses radiology reports during training to preserve instance-specific supervision while retaining image-only inference. The framework reformulates finding-conditioned grounding as multi-label segmentation and improves performance across backbones and datasets.
- Motivation: Dense voxel-level labels are costly, while routinely generated radiology reports provide semantic information about abnormalities.Reports can complement dense segmentation labels without requiring another annotation effort.
- Limitations of Existing Paradigms: Existing text-conditioned methods require a report-derived finding at inference and localize only one queried finding per forward pass.Image-only methods avoid text dependence but ignore report information.
- Proposed Method: IGRA anchors each annotated abnormality instance to its corresponding report finding during training, then removes all text-related components for deployment.The module pools an instance representation and aligns it with the corresponding finding embedding.
- Results: IGRA consistently improves Dice and hit rate across four standard 3D segmentation backbones on ReXGroundingCT.The module can be incorporated without changing the inference architecture.
- Proposed Method: The method reformulates finding-conditioned 3D grounding as multi-label volumetric segmentation by merging same-category instances.All abnormality categories can be predicted from a CT volume in a single image-only forward pass.
- Results: IGRA demonstrates zero-shot generalization to LIDC-IDRI, PleThora, and an in-house clinical dataset without dataset-specific adaptation.These evaluations support transfer beyond the primary benchmark.
2. Methodology
The paper reformulates report-conditioned grounding as text-free multi-label volumetric segmentation and preserves instance–finding correspondence during training through IGRA. Text guidance is removed at deployment, leaving an image-only segmentation model that predicts all categories jointly.
- Problem Definition: ReXGroundingCT pairs each abnormality mask with a finding sentence, but multiple same-category findings can occur in one scan.
- Problem Definition: The reformulation merges same-category findings and predicts all abnormality categories jointly with a single text-free mapping.
- Instance-Level Correspondence Preservation: IGRA preserves instance-specific correspondence by carrying a finding-assignment map alongside category-level targets.
- Instance-Level Representation: For each visible finding, the model forms a masked average-pooled instance representation from the decoder feature map before segmentation.Pooling uses full-resolution features to preserve spatial extent, while τmin = 16 excludes very small visible regions.
- Anchoring Objective: The anchoring objective minimizes cosine distance between each projected instance representation and its corresponding finding embedding.
- Training and Inference: Text, assignment maps, and the projector are used only during training; inference returns the original image-only mapping fθ.
- Anchoring Objective: Positive-only anchoring avoids false negatives from near-duplicate same-category sentences and remains effective when batches contain few valid findings.
3. Experiments
Experiments use ReXGroundingCT and evaluate a category-level multi-label 3D segmentation formulation across 14 abnormality classes. The setup combines a pretrained chest-CT encoder with a standard image-only decoder and follows the dataset’s official split.
- Implementation Details: The baseline pairs VoxTell’s pretrained ResEnc-L encoder with a randomly initialized standard U-Net decoder to avoid retaining text conditioning.
- Datasets: The official split uses 2,992 training scans and 200 validation scans, with validation and test masks exhaustively annotated by board-certified radiologists.
- Datasets: Findings sharing an ontology category within a volume are merged into one binary channel, producing 14 lesion classes for multi-label segmentation.
- Datasets: The in-house dataset includes pulmonary abnormalities that are spatially extensive and vary substantially in lesion burden.
4. Results
IGRA improves abnormality segmentation on ReXGroundingCT, generalizes across backbones and datasets, and retains image-only inference after training-time report anchoring.
- ReXGroundingCT Results: 30.93 Dice, 20.76 mIoU, 38.24 NSD, and 74.49 Hit rate were achieved in the all-findings setting.Same-category annotations were merged for category-level segmentation.
- ReXGroundingCT Results: 22.5% relative Dice improvement over TransUNet was obtained, increasing Dice from 25.25 to 30.93.IGRA also substantially improved NSD and Hit rate over the strongest image-only baseline.
- ReXGroundingCT Results: 30.29 Dice in the single-finding setting was comparable to VoxTell’s 30.43, while IGRA achieved a 72.88 Hit rate without inference-time text.All abnormality categories can be segmented directly from the CT volume.
- External and In-house Evaluation: Zero-shot evaluation showed improved cross-dataset generalization on LIDC-IDRI, PleThora, and an in-house dataset without adaptation.Qualitative comparisons highlighted closer agreement with ground-truth regions across the three datasets.
- Cross-backbone Evaluation: IGRA improved Dice across all four tested 3D backbones, with gains of +0.10, +1.92, +1.54, and +5.67 points.The largest gain was for SegResNet, whose Dice increased from 17.80 to 23.47.
- Ablation Studies: 25.65 to 30.93 Dice and 63.05 to 74.49 Hit rate followed introduction of IGRA at λ = 0.1.Performance remained relatively stable for λ ∈[0.1, 0.7], while Dice decreased to 26.47 at λ = 1.0.
- Voxel-wise Instance-Report Similarity: Voxel-wise similarity maps showed high similarity between annotated abnormality regions and their corresponding finding sentences for nodules, ground-glass opacities, and pleural effusions.The maps project decoder features and compare them with the paired sentence embedding.
5. Discussion
The method’s scope is limited by the ReXGroundingCT ontology and its reliance on paired lesion masks and finding sentences during training.
- Limitations: IGRA directly segments only the 14 abnormality categories defined by ReXGroundingCT, excluding findings outside that ontology.The paper identifies open-vocabulary abnormalities as an important extension.
- Limitations: Training uses approximately 3,000 CT scans with paired finding sentences and lesion masks, whereas larger clinical datasets often lack lesion-level annotations.The paper proposes pseudo-label generation and anatomy-aware adaptation for weaker supervision.
6. Conclusion
IGRA uses instance-specific report findings during training to improve image-only 3D chest CT segmentation, while removing text components at inference. It improves performance across architectures and datasets without requiring report dependence during deployment.
- IGRA preserves instance-specific correspondence between abnormality annotations and report findings during training despite merging same-category instances for multi-label segmentation.
- IGRA improved Dice and hit rate consistently across four standard 3D segmentation backbones, supporting its use as a model-agnostic training objective.
- 22.5%: IGRA improved Dice over the strongest image-only baseline on ReXGroundingCT while remaining comparable to VoxTell on the single-finding subset.
- IGRA generalized without adaptation to LIDC-IDRI, PleThora, and the in-house dataset.
- Text-related components are removed at inference, enabling report-guided training without report dependence during deployment.