Source-linked AI summary
MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network
Zizhao Zhang, Yuanpu Xie, Fuyong Xing, Mason McGough, Lin Yang
TL;DR
Generic computer-aided diagnosis models often lack semantically and visually meaningful explanations. MDNet creates a direct image-report mapping with report generation, image retrieval, and attention visualization, and the authors report stronger comparative performance plus state-of-the-art image-model results on CIFAR datasets.
Problem
Generic computer-aided diagnosis classification models conceal the rationale for their conclusions, limiting semantically and visually meaningful justifications.
Method
MDNet jointly models medical images and diagnostic reports with an image model, LSTM language model, improved attention, and end-to-end training.
Results
MDNet outperforms well-known image-captioning baselines on BCIDR, while its image model achieves state-of-the-art performance on two CIFAR datasets.
Takeaways & Limitations
MDNet provides report generation, symptom-based image retrieval, and attention visualization as semantic and visual justifications for diagnosis.
Takeaways & Limitations
The authors identify BCIDR’s small dataset size and call for larger pathology image-report datasets, finer attention, and whole-slide diagnosis.
Abstract
from arXiv · showhide
The inability to interpret the model prediction in semantically and visually meaningful ways is a well-known shortcoming of most existing computer-aided diagnosis methods. In this paper, we propose MDNet to establish a direct multimodal mapping between medical images and diagnostic reports that can read images, generate diagnostic reports, retrieve images by symptom descriptions, and visualize attention, to provide justifications of the network diagnosis process. MDNet includes an image model and a language model. The image model is proposed to enhance multi-scale feature ensembles and utilization efficiency. The language model, integrated with our improved attention mechanism, aims to read and explore discriminative image feature descriptions from reports to learn a direct mapping from sentence words to image pixels. The overall network is trained end-to-end by using our developed optimization strategy. Based on a pathology bladder cancer images and its diagnostic reports (BCIDR) dataset, we conduct sufficient experiments to demonstrate that MDNet outperforms comparative baselines. The proposed image model obtains state-of-the-art performance on two CIFAR datasets as well.
1. Introduction
MDNet addresses the limited interpretability of generic medical-image classification by linking images with diagnostic reports and visual attention. Its design combines image and language modeling, with experiments on bladder cancer reports and CIFAR datasets.
- Generic classification models conceal the rationale for medical diagnoses, limiting interpretable justification of their decisions.
- Medical specialists record microscopic image findings in diagnostic reports, motivating models that imitate this process for interpretability.
- MDNet reads images, generates diagnostic reports, retrieves images from symptom descriptions, and visualizes attention to justify diagnosis.
- Bladder pathology diagnosis depends on subtle changes in urothelial nuclei size and density or tissue thickening, which are difficult to discriminate accurately.
- The image model targets size-variant features through ensemble-connection for more efficient multi-scale representation integration.
- MDNet outperforms well-known image-captioning baselines on BCIDR, while its image model achieves state-of-the-art performance on two CIFAR datasets.
2. Related Work
Prior work links images and language to improve semantic descriptions and visual explanations, but medical-image applications remain limited. Joint end-to-end image-sentence modeling is identified as an open question because universal pretrained CNNs are unavailable in medicine.
- Joint image-language modeling generates semantic descriptions that make predictions more intelligible.
- A complete end-to-end trainable image-sentence model is presented as an open question because universal pretrained CNN models are unavailable for medical images.
- Attention mechanisms support alignment between natural-language words and visual features, while also encouraging visual explanations of network operation.
- Medical-image language modeling has relatively little prior work, with existing examples generating disease keywords for radiology images.
3. Image model
The image model redesigns residual-network connections to integrate multi-scale features and let the classifier weight ensemble outputs independently. This preserves direct information flow while addressing shared-classifier limitations.
- 3.1. Residual networks: Residual skip-connections provide direct information flow and help alleviate gradient vanishing as network depth increases.
- 3.1. Residual networks: A ResNet with n residual blocks can be viewed as an ensemble of 2^n relatively shallow networks, motivating explicit ensemble design.
- 3.2. Decouple ensemble network outputs: The classification module’s global average pooling and fully connected layers produce class probabilities from residual feature maps.
- 3.2. Decouple ensemble network outputs: Using one weighting function for all ensemble outputs can undermine the importance of individual features because the outputs share classifiers.
- 3.2. Decouple ensemble network outputs: Ensemble-connection decouples residual outputs by applying individual classifiers, allowing the module to determine information importance separately for each output.
- 3.2. Decouple ensemble network outputs: The design concatenates residual-block outputs into feature maps that flow directly to the classification layer in parallel.
- 3.2. Decouple ensemble network outputs: Ensemble-connection is applied between block groups where feature-map dimensions change, integrating multi-scale representations in the final convolutional layer.
4. Language modeling and network training
MDNet uses an LSTM with visual attention to connect diagnostic-report descriptions to localized image features, while AAS sharpens attention and end-to-end training combines image and language supervision. Its training duplicates samples for description-specific modeling, merges resulting image gradients, and dynamically regulates composite gradients.
- Language modeling: The LSTM models diagnostic reports by maximizing joint sentence probability and uses visual context to predict each next word.The context vector is designed to capture localized visual information from image features.
- Language modeling: Dynamic attention weights extract image features supporting word prediction and produce maps of where the network captures visual information.The original mechanism often spreads attention smoothly across most of the image, complicating training and interpretation.
- Language modeling: AAS adds supervision based on diagnostic conclusions to learn class-specific, localized feature embeddings that improve alignment between image regions and description words.The embedding uses a class-selected weight vector applied to convolutional feature maps.
- Language modeling: Description-specific LSTM functions share parameters, while duplicated image-report samples target different image-feature types and merge their gradients during backpropagation.This design lets each description mine discriminative information while remaining part of complete report generation; all operations are end-to-end trainable.
- Network training: MDNet jointly optimizes image, language, and AAS parameters using negative log-likelihood losses, with image-model updates receiving gradients from both modules.The training scheme dynamically regulates the two gradients and controls the language-loss gradient scale through β and η.
5. Experimental Results
Experiments evaluate EcNet on CIFAR recognition and MDNet on BCIDR report generation, symptom-based retrieval, and attention-based interpretability. MDNet outperforms comparative baselines, while EcNet achieves strong CIFAR error rates with efficient feature utilization.
- Image recognition on CIFAR: The CIFAR experiments compare two EcNet variants with ResNet and WRN architectures using error rate and training curves.The study uses CIFAR-10 and CIFAR-100 and selects baseline architectures by depth and width to trade off memory usage and training efficiency.
- Image recognition on CIFAR: EcNet-56-12 achieves 4.43% error on CIFAR-10 and 19.94% on CIFAR-100 with 8M parameters.These results outperform similarly sized WRN-40-4 and ResNet+-1001 variants in the reported comparison.
- Interpret model prediction: Sentence-guided attention maps associate predicted report words with specific image regions and are reported to focus on informative areas.Figure 5 highlights sharp attention on urothelial neoplasms used to diagnose carcinoma type.
- Interpret model prediction: Class-specific attention maps show strong correspondence with pathologist annotations and activate mainly in urothelial regions without region-level training annotations.The maps largely ignore stromal or background regions.
- Diagnostic report generation: MDNet significantly improves DCA, with smaller standard deviation, and improves most other reported metrics over comparative baseline models.Fine-tuning pre-trained CNNs generally helps but is described as more unstable because of higher standard deviation.
- Symptom description based image retrieval: For symptom-based retrieval, MDNet is more accurate than baselines at mapping image descriptions to images with the correct diagnostic conclusion.The evaluation reports correct conclusion recall Cr@k over five folds; joint modeling and fine-tuning further improve the mapping quality.
6. Discussion
The discussion examines MDNet’s gradient-balancing optimization, image–language interaction, and constraints from the small BCIDR dataset. It reports that weighted gradient averaging supports training stability and that language-model diagnostic accuracy depends strongly on the image model.
- Optimization: A sigmoid-like schedule changes β from 0 to 1 during training, allowing image-model training to dominate initially before semantic knowledge is introduced.
- Optimization: Weighted averaging of the two module gradients is necessary because simply scaling one gradient increases training instability.The summed gradient with respect to θ_D can grow larger when the 1−β term is removed.
- Optimization: ∼4% DCA score decrease occurs for the language model without averaging.
- Small dataset and regularization: The small BCIDR dataset increases the possibility of overfitting, making effective regularization necessary despite the use of regularization techniques and cross-validation.
- Multimodal mapping: The language model’s DCA strongly relies on the DCA of its corresponding image model across all trained models.
- Multimodal mapping: MDNet’s multimodal mapping uses semantic information from diagnostic reports to promote knowledge sharing between image and language models, yielding better mapping quality and more accurate prediction.
7. Conclusion and Future Work
MDNet establishes a direct multimodal mapping between medical images and diagnostic reports, making diagnosis semantically and visually interpretable. The authors identify larger datasets, finer attention localization, and whole-slide diagnosis as future directions.
- 7. Conclusion and Future Work: MDNet generates diagnostic reports and corresponding network attention to make diagnosis and decision-making semantically and visually interpretable.
- 7. Conclusion and Future Work: Future work includes building and testing large-scale pathology image-report datasets.
- 7. Conclusion and Future Work: Future work includes generating finer attention for small biomarker localization and applying MDNet to whole-slide diagnosis.