Source-linked AI summary
Boot-and-Feedback Framework for Generalist-Expert Model Collaboration in Breast Ultrasound Diagnosis
Ming Cheng, Hongyu Sun, Zhaolin Chen, Jun Liu, Hossein Rahmani, Qiuhong Ke
TL;DR
Breast ultrasound diagnosis requires reliable and interpretable assistance, but MLLMs can generate domain-misaligned descriptions that mislead expert models. BooF guides MLLMs with BI-RADS terminology and vision-expert priors, then uses attention-gated multimodal feedback to filter textual noise. Across two public BUS datasets, BooF outperforms state-of-the-art methods in diagnostic accuracy and interpretability.
Problem
BUS diagnosis remains operator-dependent, while existing deep models lack structured clinical reasoning and MLLMs can produce hallucinated, domain-misaligned descriptions.
Method
BooF uses BI-RADS lexicon alignment and vision-expert prediction priors to guide MLLM descriptions, then fuses them with visual features through attention-gated multimodal feedback.
Results
BooF outperforms state-of-the-art methods in diagnostic accuracy and interpretability across two public BUS datasets.
Takeaways & Limitations
BooF provides clinically grounded BI-RADS-aligned rationales alongside improved BUS diagnostic performance.
Abstract
from arXiv · showhide
Breast ultrasound (BUS) is widely used for breast cancer diagnosis yet remains operator-dependent. While deep learning shows promise, ensuring diagnostic reliability and interpretability is challenging. Recent Multimodal Large Language Models (MLLMs) often generate spurious descriptions due to limited domain knowledge, which mislead downstream expert models and compromise clinical validity. To address these challenges, we propose the Boot-and-Feedback (BooF) model collaboration framework for synergistic MLLM-expert interaction. Specifically, in the Boot Stage, the MLLM is guided by the BI-RADS lexicon and preliminary benign-malignant vision-expert predictions, enabling it to transfer general reasoning to BUS analysis while avoiding hallucinations. Subsequently, the Feedback Stage integrates these descriptions with visual features via a lightweight Attention-Gated Cross-Modality Fusion Module. This allows the expert to leverage textual feedback while adaptively filtering noise. Extensive experiments on multiple BUS datasets demonstrate that BooF substantially outperforms state-of-the-art methods in terms of diagnostic accuracy and interpretability.
1. INTRODUCTION
Breast ultrasound diagnosis needs CAD systems that combine accuracy with interpretable clinical reasoning, but existing models face operator variability, limited structured reasoning, and MLLM hallucinations. BooF addresses these issues through bidirectional MLLM–expert collaboration using BI-RADS guidance, vision-expert priors, and attention-gated feedback.
- BUS interpretation remains operator-dependent, creating variability across clinical settings and motivating accurate, interpretable CAD assistance.
- Deep learning models can perform competitively but lack structured clinical reasoning, while MLLMs generate readable descriptions yet remain vulnerable to hallucinations and domain misalignment.
- MLLM descriptions can conflict with expert predictions because of insufficient domain knowledge and unidirectional interaction, undermining clinical reliability.
- BooF enables bidirectional collaboration by aligning MLLM outputs with BI-RADS terminology and incorporating preliminary benign–malignant vision-expert predictions.
- The feedback stage combines enhanced descriptions with ultrasound features through attention-gated fusion to filter semantic noise and improve diagnosis.
- BooF outperforms state-of-the-art methods in diagnostic accuracy and interpretability across two public BUS datasets.
2. METHODS
BooF adapts MLLM reasoning to BUS through BI-RADS descriptions and calibrated vision-expert priors, then feeds the descriptions into a multimodal expert. Attention-gated fusion weights textual attributes against image features for malignancy prediction while reducing hallucination noise.
- Directly applying a general MLLM to BUS diagnosis can produce hallucinatory reasoning and degrade downstream vision-language performance without domain adaptation.
- BI-RADS alignment replaces direct benign–malignant prompting with standardized descriptions of five lesion properties grounded in imaging characteristics.
- VEPP supplies a calibrated benign–malignant prior only when confidence exceeds a validation-selected threshold, while instructing the MLLM to identify contradictions.
- VEPP narrows the reasoning space while preserving independence, reducing hallucination-prone outputs and allowing the MLLM to challenge erroneous priors.
- The vision-language expert jointly encodes BI-RADS descriptors and BUS image features to refine malignancy prediction and reinforce concordance between descriptions and diagnoses.
- AGCFM computes relevance between image features and textual attributes, selectively fuses the resulting representations, and produces malignancy logits optimized with cross-entropy.The fused representation adds projected text information to the image representation before classification.
3. EXPERIMENTS
Experiments on BUS-BRA and BUSI evaluate BooF through component ablations and comparisons with state-of-the-art methods. BooF consistently improves diagnostic performance while providing BI-RADS-aligned interpretability.
- Experimental Setup: The evaluation uses BUS-BRA and BUSI, with BUS-BRA ablations employing RadBERT alongside MedViT and ResNet-50 visual encoders.BUS-BRA contains benign and malignant images with BI-RADS annotations, while BUSI contains benign and malignant images.
- Boot and Feedback Ablation: Naively adding MLLM-generated descriptions is unreliable, with ResNet-50 AUC falling below the vision-only baseline.The text can be spurious or clinically inconsistent and misguide the expert model.
- Boot and Feedback Ablation: BI-RADS Lexicon Alignment produces consistent performance gains, while Vision Expert Prediction Priors Guidance yields the largest improvements, particularly in Recall and AUC.The prior guidance grounds MLLM reasoning in benign–malignant vision-expert predictions and suppresses hallucinations.
- Boot and Feedback Ablation: Attention-gated multimodal fusion provides an additional stable boost by integrating enhanced descriptions with visual features while filtering semantic noise.This constitutes the feedback stage of the BooF configuration.
- Boot and Feedback Ablation: Across both backbones, complete BooF improves over the vision-only baseline by approximately 4.8%–6.1% in AUC, 5.7%–7.4% in Accuracy, and 14%–16% in Recall.These are absolute improvements reported across the two backbones.
- Comparison with State-of-the-Art Methods: On BUSI, BooF achieves AUC 0.959, Specificity 0.962, and F1-score 0.885; on BUS-BRA, it achieves AUC 0.976, Recall 0.903, and F1-score 0.899.BooF is reported to outperform all compared methods and also provides BI-RADS-aligned rationales for radiologist verification.
4. CONCLUSION
The conclusion presents BooF as a dynamic, bidirectional framework combining MLLMs with domain-specific expert models for breast ultrasound analysis. Experiments on two public datasets support improvements in diagnostic accuracy and interpretability, while future work may extend the approach to multi-round interactions.
- Framework Contribution: BooF enables dynamic bidirectional interaction between general MLLMs and domain-specific expert models.The framework uses boot-stage guidance followed by feedback-stage multimodal attention fusion.
- Framework Contribution: BI-RADS Lexicon Alignment and Vision Expert Prediction Priors Guidance support the boot stage, followed by multimodal attention fusion in the feedback stage.These components are presented as the framework's main mechanisms for improving breast ultrasound analysis.
- Conclusion: Experiments on two public datasets demonstrate that BooF surpasses state-of-the-art methods in diagnostic precision and clinically relevant decision rationales.The conclusion describes the framework as highly adaptable to other specialized medical classification tasks.
- Future Work: Future research could extend BooF to multi-round iterative interactions to further enhance diagnostic performance and robustness.This is identified as a future direction rather than a demonstrated result.