Source-linked AI summary
METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens
Zhanyu Wang, Lingqiao Liu, Lei Wang, Luping Zhou
TL;DR
Radiology report generation must handle fine-grained abnormalities, while existing work commonly uses single-expert frameworks. METransformer introduces multiple interacting expert tokens, complementary representations, and metric-based voting, achieving promising results on two benchmarks and retaining a framework that can incorporate further advances.
Problem
Existing radiology report generation methods commonly follow single-expert encoder-decoder designs, although intricate cases motivate multi-specialist consultation and fine-grained abnormalities are difficult to localize.
Method
METransformer inserts multiple learnable expert tokens into transformer encoder and decoder stages, encourages complementary representations with orthogonal loss, and selects among candidate reports using metric-based expert voting.
Results
METransformer shows promising performance on IU-Xray and MIMIC-CXR, with CIDEr 0.362 on MIMIC-CXR versus 0.299 for MSAT.
Takeaways & Limitations
The framework provides ensemble-like benefits with more efficient computation and can incorporate advances from existing single-expert methods.
Takeaways & Limitations
The framework remains basic and is limited by not yet integrating medical domain knowledge from single-expert methods; clinical efficacy metrics apply only to MIMIC-CXR.
Abstract
from arXiv · showhide
In clinical scenarios, multi-specialist consultation could significantly benefit the diagnosis, especially for intricate cases. This inspires us to explore a "multi-expert joint diagnosis" mechanism to upgrade the existing "single expert" framework commonly seen in the current literature. To this end, we propose METransformer, a method to realize this idea with a transformer-based backbone. The key design of our method is the introduction of multiple learnable "expert" tokens into both the transformer encoder and decoder. In the encoder, each expert token interacts with both vision tokens and other expert tokens to learn to attend different image regions for image representation. These expert tokens are encouraged to capture complementary information by an orthogonal loss that minimizes their overlap. In the decoder, each attended expert token guides the cross-attention between input words and visual tokens, thus influencing the generated report. A metrics-based expert voting strategy is further developed to generate the final report. By the multi-experts concept, our model enjoys the merits of an ensemble-based approach but through a manner that is computationally more efficient and supports more sophisticated interactions among experts. Experimental results demonstrate the promising performance of our proposed model on two widely used benchmarks. Last but not least, the framework-level innovation makes our work ready to incorporate advances on existing "single-expert" models to further improve its performance.
1. Introduction
METransformer reframes radiology report generation as multi-expert joint diagnosis, addressing the difficulty of locating fine-grained abnormalities and extending single-expert encoder-decoder methods. It uses complementary learnable expert tokens across a transformer encoder and decoder, with ensemble-like benefits and promising benchmark performance.
- Radiology report generation aims to produce free-text descriptions from radiographs while reducing radiologists’ manual workload.
- Fine-grained abnormalities can occupy small, arbitrary image regions, making consistent attention during report generation difficult.
- Encoder expert tokens interact with visual and other expert tokens, use higher-order attention, and receive orthogonal-loss guidance toward complementary image representations.
- Decoder expert representations guide word and visual embeddings to produce multiple candidate reports, followed by metric-based expert voting for the final report.
- METransformer introduces multiple learnable expert tokens into the transformer encoder and decoder to model multi-specialist joint diagnosis.
- The approach shows promising performance on IU-Xray and MIMIC-CXR while offering ensemble-like behavior with reduced training parameters and improved training efficiency.
2. Related Work
Related work situates radiology report generation within image captioning, where encoder-decoder architectures and attention mechanisms are common. Existing decoder enhancements can also be incorporated into the proposed framework.
- Natural image captioning generates a sentence from an image, commonly using CNN encoders and recurrent or Transformer decoders with attention modules.
- The memory-enhanced decoder used in prior work can also be applied to the METransformer framework for further performance improvement.
3. Method
METransformer uses multiple learnable expert tokens throughout an encoder-decoder architecture to capture complementary fine-grained visual information and generate diverse reports. Expert-specific decoding and metric-based voting produce the final diagnostic report.
- Expert Transformer Encoder: The encoder combines visual and expert tokens, using linear self-attention followed by higher-order bilinear attention to enhance fine-grained image representations.Expert tokens interact with visual tokens and with one another before bilinear attention further models expert–visual interactions.
- Expert Transformer Encoder: An orthogonal loss encourages different learnable expert tokens to attend to different image regions and capture complementary representations.The expert tokens are appended to the visual-token sequence and trained to reduce overlap among their embeddings.
- Expert Transformer Decoder: The decoder sends enhanced expert-token embeddings into separate expert pathways, where adjustment blocks regulate word and visual embeddings for expert-specific report generation.Each expert token influences the report generation process, enabling discrepant candidate reports from orthogonally trained experts.
- Expert Voting Strategy: The model produces M candidate reports and selects the report with the highest CIDEr-based consistency score against the other experts.The voting strategy uses NLG metric scores and selects one complete report rather than fusing outputs word by word.
4. Experiments
Experiments evaluate METransformer on IU-Xray and MIMIC-CXR using natural-language and clinical-efficacy metrics, comparisons with prior models, ablations, and qualitative visualizations. Results show consistent advantages over single-expert approaches, with expert-token design and voting further examined.
- Experimental Settings: Evaluation uses IU-Xray and MIMIC-CXR, with BLEU-4, METEOR, ROUGE-L, and CIDEr for generated-report quality.Clinical efficacy is additionally measured on MIMIC-CXR using CheXpert labels across 14 thoracic-disease and support-device categories.
- Main Results: METransformer consistently outperforms single-expert baselines on both datasets and achieves the best performance across all metrics on MIMIC-CXR.Its MIMIC-CXR CIDEr score is 0.362, compared with 0.299 for MSAT.
- Ablation Study: Ablations isolate Bilinear-attention Encoder, Expert Tokens, Orthogonal Loss, and Expert Voting against a ViT-plus-bilinear-attention baseline.With multiple expert tokens, final predictions average experts’ word probabilities, while comparison of ablations evaluates the proposed voting strategy.
- Ablation Study: Increasing expert-token count generally improves BLEU-4 and CIDEr, but gains stop beyond a threshold; IU-Xray CIDEr decreases from seven to nine tokens.The study varies the number of expert tokens over {1, 3, 5, 7, 9}.
- Main Results: The ensemble comparison evaluates random-initialization, multiple-decoder, and stochastic model-averaging ensembles using Expert Voting on IU-Xray.These comparisons test whether METransformer’s multi-expert design offers an alternative to conventional ensemble constructions.
- Qualitative Analysis: Visualizations show distinct expert-token attention regions and stronger alignment between generated medical terms, image locations, and radiologist-written descriptions.An example includes heart anomalies detected by METransformer but missed by the baseline.
5. Conclusions
METransformer introduces multiple learnable expert tokens into a transformer-based framework for radiology report generation, following the concept of multi-specialist consultation. The framework shows promising performance while remaining open to enhancement with medical domain knowledge.
- METransformer introduces multiple learnable expert tokens into a transformer-based framework for radiology report generation.
- The framework follows the concept of multi-specialist consultation to improve generated report quality.
- METransformer is a basic framework that could be enhanced by integrating medical domain knowledge from single-expert methods.