Source-linked AI summary
Competence-based Multimodal Curriculum Learning for Medical Report Generation
Fenglin Liu, Shen Ge, Yuexian Zou, Xian Wu
TL;DR
Medical report generation is challenged by serious data bias and limited medical data, which complicate accurate abnormality description. CMCL uses competence-based multimodal curriculum learning to assess instance difficulty and select suitable training batches progressively. Across IU-Xray and MIMIC-CXR, it consistently improves baseline performance under most metrics and receives favorable clinician evaluations.
Problem
Medical report generation faces serious visual and textual data bias alongside limited medical data, making accurate abnormality description difficult.
Method
CMCL evaluates visual and textual difficulty from multiple metrics, estimates model competence, and progressively selects appropriate training curricula from easy to complex samples.
Results
CMCL consistently boosts baseline performance under most metrics and clinicians find its reports more fluent and more accurate in describing abnormalities.
Takeaways & Limitations
CMCL can be incorporated into existing models without additional parameters and with only a small training-data-pipeline modification.
Takeaways & Limitations
The framework does not consider language-difficulty metrics such as sentence length, n-gram rarity, NER, or POS tagging.
Abstract
from arXiv · showhide
Medical report generation task, which targets to produce long and coherent descriptions of medical images, has attracted growing research interests recently. Different from the general image captioning tasks, medical report generation is more challenging for data-driven neural models. This is mainly due to 1) the serious data bias and 2) the limited medical data. To alleviate the data bias and make best use of available data, we propose a Competence-based Multimodal Curriculum Learning framework (CMCL). Specifically, CMCL simulates the learning process of radiologists and optimizes the model in a step by step manner. Firstly, CMCL estimates the difficulty of each training instance and evaluates the competence of current model; Secondly, CMCL selects the most suitable batch of training instances considering current model competence. By iterating above two steps, CMCL can gradually improve the model's performance. The experiments on the public IU-Xray and MIMIC-CXR datasets show that CMCL can be incorporated into existing models to improve their performance.
1 Introduction
Medical report generation is clinically useful but difficult because limited, biased multimodal data causes models to miss rare abnormalities and generate errors. CMCL addresses this by progressively selecting training instances according to multimodal difficulty and model competence.
- Medical report generation can assist radiologists because manual reporting is time-consuming for experienced clinicians and error-prone for inexperienced ones.
- Visual and textual biases cause normal images, regions, and sentences to dominate training data, while abnormal findings remain underrepresented.
- CMCL assesses visual and textual complexity, then selects training samples suited to the model’s current competence in an easy-to-hard progression.
- Curriculum learning helps existing models use limited medical data more effectively and alleviate data bias by progressing from easy samples to complex ones.
- CMCL improves existing baseline models on most metrics across two public datasets without adding parameters and with only a small training-pipeline modification.
2 Related Work
Medical report generation extends image captioning from single-sentence description to detailed, clinically accurate reports. Existing approaches face multimodal data bias, while single-difficulty curriculum methods cannot represent both visual and textual complexity.
- Medical reports must cover key findings, accurately describe abnormalities with details, and identify potential diseases such as effusion and pneumothorax.
- CNN-HLSTM-based report generation models inherit data-bias and limited-data challenges, making abnormality description particularly difficult.
- Curriculum learning trains models progressively from easy samples to more complex ones and has been studied in natural language processing and computer vision.
- Single-difficulty unimodal curricula are unsuitable because medical report generation combines visual medical images and textual reports with distinct complexities.
- The paper introduces competence-based multimodal curriculum learning with multiple difficulty metrics.
3 Framework
Typical models randomly batch image-report pairs after CNN encoding, treating all training instances equally. CMCL instead ranks instances by multiple difficulty metrics and selects the batch with the highest current-model perplexity.
- Typical medical report generation uses a CNN image encoder followed by an LSTM-like sequence generator, with randomly shuffled training batches.
- CMCL builds training batches selectively rather than treating all training instances equally.
- For multimodal data, CMCL ranks training instances independently under multiple metrics and creates one candidate batch per metric.
- At each step, CMCL computes each candidate batch’s perplexity under the current model and trains on the batch with highest perplexity.
4 Difficulty Metrics
CMCL measures difficulty from visual and textual perspectives because abnormalities are difficult to capture and describe. Its metrics combine heuristic properties with model confidence, while textual difficulty intentionally excludes some language-complexity measures.
- Difficulty basis: CMCL defines difficulty metrics around the challenge of accurately capturing and describing abnormalities in medical reports.
- Visual Difficulty: Visual difficulty includes heuristic and model-based measures derived from ResNet-50 representations and abnormality classification probabilities.
- Visual Difficulty: The entropy of the abnormality-probability distribution serves as a model-confidence measure indicating whether an image is easy to classify.
- Textual Difficulty: Textual heuristic difficulty counts abnormal sentences, treating sentences containing “no”, “normal”, “clear”, or “stable” as normal.
- Textual Difficulty: Textual difficulty excludes sentence length, n-gram rarity, named-entity recognition, and part-of-speech tagging because the framework focuses on capturing and describing abnormalities.
5 Approach
CMCL extends competence-based curriculum learning to medical report generation by modeling multiple difficulty dimensions and adaptively selecting training curricula. It progressively trains models from easier to harder samples while targeting the under-learned curriculum.
- CMCL computes four difficulty measures, sorts the training data separately for each measure, and initializes model competence for each perspective.
- Medical report generation requires multiple difficulty metrics because each training instance combines an image with its corresponding report.
- Unlike simply fusing difficulty metrics or cycling curricula in a fixed order, CMCL adaptively chooses the most appropriate curriculum for each training step.
- At each training step, CMCL selects the curriculum with the highest perplexity, indicating that the model has learned it less well.
- The selected curriculum supplies a batch from the portion of its sorted dataset allowed by current competence, after which competence and perplexity are updated.
6 Experiment
The experiments use the IU-Xray and MIMIC-CXR public chest X-ray datasets, with established dataset splits and preprocessing procedures. Figure 3 illustrates the multiple-difficulty curriculum algorithm.
- The evaluation uses two public datasets: the widely used IU-Xray benchmark and the large-scale MIMIC-CXR dataset.
- Figure 3 illustrates Algorithm 2, the proposed multiple-difficulty curriculum learning procedure.
- IU-Xray contains 7,470 chest X-ray images paired with 3,955 radiology reports.
- MIMIC-CXR contains 377,110 chest X-ray images and 227,835 radiology reports from 64,588 patients.
- IU-Xray is split randomly into 70%-10%-20% training, validation, and testing sets, while MIMIC-CXR uses its official splits.
6.2 Baselines
The study evaluates CMCL with image-captioning and medical-report-generation baselines spanning encoder-decoder, attention, hierarchical, and co-attention architectures.
- The baselines include three models originally designed for image captioning and three models originally designed for medical report generation.
- NIC uses a CNN image encoder and an RNN decoder to generate target sentences.
- Spatial-Attention focuses visual processing on relevant image regions through visual attention over hidden states.
- Adaptive-Attention determines when visual attention is needed for predicting words with little or no visual content.
- CNN-HLSTM generates sentence-level topics with a paragraph LSTM and then produces each sentence with a sentence LSTM.
- HLSTM+att+Dual uses attention plus separate Normal and Abnormal LSTMs to generate more accurate normal and abnormal sentences.
- Co-Attention jointly attends to images and predicted semantic tags, combining visual and semantic information.
6.3 Metrics and Settings
The experiments use standard automatic metrics, preserve baseline architectures and parameters, and compare CMCL-equipped models across datasets and clinical evaluation settings. Reported results show gains on most metrics and competitive state-of-the-art performance.
- Metrics: Performance is measured with BLEU, METEOR, and ROUGE-L using a standard evaluation toolkit.
- Settings: Baseline inner structures and original parameter settings remain unchanged so the experiments isolate the training-paradigm modification.
- Automatic Evaluation: Across IU-Xray and MIMIC-CXR, CMCL-equipped baselines improve over most automatic-evaluation metrics.
- Human Evaluation: Two professional clinicians evaluate 100 randomly selected reports generated by CNN-HLSTM and Co-Attention with and without CMCL.
- Automatic Evaluation: CMCL applied to Co-Attention achieves results competitive with selected state-of-the-art models on major metrics.
6.5 Human Evaluation
Human evaluation and ablation analyses indicate that CMCL improves report quality, especially abnormality descriptions, by combining difficulty metrics with dynamic curriculum selection.
- Human Evaluation: All invited clinicians found CMCL reports more fluent and more accurate in describing abnormalities than baseline reports.Clinicians evaluated 100 randomly selected reports generated by CNN-HLSTM, Co-Attention, and their CMCL-equipped versions without knowing the generating model.
- Difficulty Metrics: Each individual difficulty metric improves baseline performance, while model-confidence metrics outperform heuristic metrics in both visual and textual difficulty.Textual difficulty metrics outperform corresponding visual metrics, suggesting textual bias is particularly important for report generation.
- Difficulty Metrics: Progressively incorporating difficulty metrics continuously increases performance, indicating that the metrics contribute complementary improvements.The full combination unites gains from different difficulty perspectives.
- Route Strategy: Direct fusion, random curricula, and sequential curricula all improve performance, but each performs worse than CMCL’s dynamically selected route.These results support the practical robustness of curriculum strategies while favoring dynamic selection at each training step.
- Qualitative Analysis: Qualitative examples show structured reports aligned with ground truth and accurate descriptions of abnormalities such as costophrenic blunting and scoliosis.The examples are presented as evidence that gradual easy-to-hard training helps alleviate data bias.
7 Conclusion
CMCL addresses medical report generation data bias by assessing four visual and textual difficulty measures and training models through increasingly complex samples. Experiments show consistent baseline improvements under most metrics and report effectiveness and generalization.
- 7 Conclusion: CMCL assesses four training-sample difficulties from visual and textual complexity, producing four curricula for easy-to-hard training.The framework selects appropriate curricula as model training progresses.
- 7 Conclusion: CMCL consistently boosts baseline performance under most metrics, demonstrating effectiveness and generalization across existing medical report generation systems.The framework is designed to use limited medical data more efficiently while alleviating data bias.
Ethical Considerations
The work focuses on helping existing medical report generation systems use limited medical data more efficiently while alleviating data bias through progressively harder training samples.
- Ethical Considerations: The framework targets existing medical report generation systems and trains them from simple reports toward harder reports containing rare and diverse abnormalities.This progression is described as similar to a radiologist’s learning curve.