Source-linked AI summary
Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation
Mingjie Li, Fuyu Wang, Xiaojun Chang, Xiaodan Liang
TL;DR
Medical report generation requires fine-grained, medically coherent descriptions, while existing global-image and corpus-only methods can overlook focused abnormalities and reproduce generic normal-region language. ASGK combines internal visual fusion and external medical linguistic signals with graph encoding and GPT-inspired decoding, and experiments on CX-CHR and COV-CTR report improvements in tag classification and report generation over existing methods.
Problem
Existing methods treat global image regions equally and train on dataset corpora alone, although radiologists focus on small abnormal regions and use external medical knowledge.
Method
ASGK uses internal fusion visual features, external medical linguistic information, a medical tag graph encoder, and a GPT-based natural language decoder.
Results
ASGK outperforms existing methods on report generation and tag classification across the CX-CHR and COV-CTR datasets.
Takeaways & Limitations
Auxiliary signals support robust and accurate medical report generation while transferring medical knowledge between visual encoding and language decoding.
Abstract
from arXiv · showhide
Beyond the common difficulties faced in the natural image captioning, medical report generation specifically requires the model to describe a medical image with a fine-grained and semantic-coherence paragraph that should satisfy both medical commonsense and logic. Previous works generally extract the global image features and attempt to generate a paragraph that is similar to referenced reports; however, this approach has two limitations. Firstly, the regions of primary interest to radiologists are usually located in a small area of the global image, meaning that the remainder parts of the image could be considered as irrelevant noise in the training procedure. Secondly, there are many similar sentences used in each medical report to describe the normal regions of the image, which causes serious data bias. This deviation is likely to teach models to generate these inessential sentences on a regular basis. To address these problems, we propose an Auxiliary Signal-Guided Knowledge Encoder-Decoder (ASGK) to mimic radiologists' working patterns. In more detail, ASGK integrates internal visual feature fusion and external medical linguistic information to guide medical knowledge transfer and learning. The core structure of ASGK consists of a medical graph encoder and a natural language decoder, inspired by advanced Generative Pre-Training (GPT). Experiments on the CX-CHR dataset and our COVID-19 CT Report dataset demonstrate that our proposed ASGK is able to generate a robust and accurate report, and moreover outperforms state-of-the-art methods on both medical terminology classification and paragraph generation metrics.
1 Introduction
Medical report generation must connect fine-grained visual abnormalities with coherent medical language, but existing global-image and corpus-only approaches can treat clinically important regions as noise and reinforce generic normal-region sentences. ASGK addresses these difficulties with internal and external auxiliary signals, a medical graph encoder, a GPT-inspired decoder, and evaluation on two datasets.
- Motivation: Medical report generation requires fine-grained, semantically coherent descriptions that accurately use medical terminology and domain knowledge.The task differs from natural image captioning because reports must satisfy medical commonsense and logic.
- Motivation: Existing methods use the global image and dataset corpora alone, giving small attention regions equal treatment with other regions and potentially reinforcing irrelevant or repetitive normal-region content.Radiologists inspect abnormal regions after browsing the global image and draw on external medical knowledge and experience.
- Approach: ASGK introduces internal fusion visual features and external medical linguistic information to guide graph encoding, medical knowledge learning, and visual-language transfer.The internal signal attends auxiliary region features to global visual features, while the external signal comes from a large-scale medical textbook.
- Evaluation: COV-CTR is a new COVID-19 CT report dataset constructed from diagnostic assessments by three radiologists with more than five years of experience.The dataset is introduced to evaluate ASGK's robustness and generalization ability.
- Evaluation: Experiments on CX-CHR and COV-CTR show improvements in both medical tag classification and report generation, with ablations examining how ASGK works.The evaluation uses CIDER-D, ROUGE-L, and BELU metrics.
- Approach: The method combines a medical tag graph encoder with GPT-based natural language decoding to generate accurate and robust medical reports.The graph encoder transfers input features into higher-level information, while the decoder produces the report.
2 Related Work
Related work develops visual captioning and medical image analysis through encoder-decoder, attention, graph, and self-supervised approaches, while pretrained language models provide transferable linguistic knowledge. These directions motivate auxiliary-signal use across visual and medical report generation.
- Visual Captioning and Medical Report Generation: Visual captioning models summarize images or videos in sentences or topic-related paragraphs, using neural encoder-decoder frameworks and attention mechanisms.Knowledge graphs have also been explored to exploit detected nodes and their relationships.
- Medical Image Analysis with Auxiliary Signals: Medical image analysis increasingly uses self-supervised learning because medical images are difficult to access and annotate.Self-supervised methods design proxy tasks that provide auxiliary training signals.
- Language Model Pre-training: Pretrained language models such as ELMO, BERT, XLNet, and GPT-2 learn syntactic and semantic information through unsupervised learning and transfer it to downstream tasks.This establishes language-model pretraining as a source of auxiliary linguistic knowledge for captioning.
3 Approach
ASGK combines a medical graph encoder with a natural language decoder, using internal visual signals and external medical text to guide medical knowledge learning and report generation. Its graph represents detected medical tags and learned relationships, while the decoder produces semantically aligned reports.
- Medical graph encoder: ASGK encodes input features into a medical tag graph and decodes the resulting high-level information into medical reports.Graph nodes represent detected medical tags, including abnormal and normal terminologies.
- Auxiliary signals: The model uses internal fusion visual features and external medical linguistic information as auxiliary signals for graph encoding and medical knowledge learning.External medical text is collected into a large-scale medical textbook for pretraining.
- Medical graph encoder: The medical graph represents node features with tag-classification probabilities and learns edge weights automatically through attention because ground-truth edge information is unavailable.The graph is further incorporated with prior medical knowledge and refined using multi-head self-attention.
- Natural language decoder: The GPT-inspired natural language decoder uses masked multi-head self-attention over report tokens and produces a distribution over the vocabulary for next-token prediction.The decoder consists of three blocks and uses pretrained word and position embeddings.
- Auxiliary signals: Internal visual signals are formed by thresholding DenseNet-121 heat maps to select connected auxiliary regions, extracting attended-region features, and fusing them with global features.The fused signals support medical graph encoding and bridge visual and linguistic information.
- Auxiliary signals: Focal loss is used to balance medical-tag deviations, with α set to 0.25 and γ set to 2 for the task.The focal-loss branch uses the label, prediction probability, and a modulating factor controlled by γ.
4 Experiments
Experiments on CX-CHR and COV-CTR evaluate ASGK using report-generation, tag-classification, and human-assessment procedures. Results show benefits from auxiliary region features, external medical knowledge, and focal loss, with ASGK generally outperforming comparison methods.
- Datasets: CX-CHR contains 45,598 images from 35,609 patients with 173 medical tags, while COV-CTR contains 728 images with 68 tags.Both datasets pair medical images with Chinese diagnostic reports and distinguish abnormal from normal terminologies.
- Evaluation: ASGK is evaluated with AUC, CIDER-L, ROUGE-L, BELU, and human comparisons using randomly selected test samples.The study also compares generated reports from ASGK, CoAtt, and Vision-Bert in human evaluation.
- Automatic Evaluation: On CX-CHR, ASGK increases CIDER scores by 51.0, 35.0, 39.5, 22.1, and 22.7 over comparison models, while slightly underperforming CoAtt on ROUGE-L.ASGK also outperforms other baselines on COV-CTR.
- Medical Tags Classification: ASGK outperforms the baseline on medical-tag classification across both datasets, with internal auxiliary signals promoting graph encoding and tag prediction.The classification evaluation includes both normal and abnormal terminologies.
- Human Evaluation: Radiologists preferred ASGK reports in 20% of CX-CHR cases and 27% of COV-CTR cases, while judging 53% and 22% of cases equivalent, respectively.The human evaluation used 200 randomly selected images and three radiologists.
- Ablation Studies: Internal auxiliary learning improves automatic metrics by 15.6%, 1.4%, and 0.6% and classification accuracy by 4.5% on CX-CHR.The authors associate these gains with focusing on abnormal regions, which provide richer visual features.
- Ablation Studies: Without focal loss, AUC decreases by 0.9% and 0.7% on the two datasets, indicating its role in addressing normal-label dominance.The paper states that images typically contain three to five normal tags and only a few abnormal terminologies.
- Ablation Studies: External auxiliary pretraining raises automatic metrics from 289.7% to 317.2% and from 59.1% to 66.9% on the two datasets, but classification accuracy rises only 0.6% and 1.8%.The authors report that external signals mainly improve the decoder’s medical phrasing and semantic coherence.
5 Conclusions and Future Work
The paper presents ASGK as a radiologist-inspired encoder-decoder for fine-grained, semantically coherent medical reports. It concludes that auxiliary signals improve report generation and tag classification across two datasets, while identifying broader auxiliary-signal-guided captioning as future work.
- Conclusions: ASGK mimics radiologists’ working patterns to generate fine-grained and semantically coherent medical reports.The approach uses auxiliary signals to guide knowledge encoding and natural-language decoding.
- Conclusions: Experiments show that ASGK outperforms existing methods on report generation and tag classification across two medical datasets.The paper also investigates auxiliary-region cropping, textbook knowledge use, and how the signals function.
- Future Work: Future work will develop a general captioning framework guided by auxiliary signals to encode and decode general-corpus knowledge.
Broader Impacts
The work presents medical report generation as clinically meaningful, particularly during pandemics, and proposes an anthropomorphic model that mimics radiologists using auxiliary signals.
- Robust and accurate medical report generation could reduce doctors’ burden and help people understand their health status more accurately.
- The proposed model mimics radiologists’ working patterns by acquiring easily accessed auxiliary signals for medical report generation.
- The approach may guide researchers with limited medical image resources toward unsupervised learning for acquiring auxiliary signals.
- The authors identify the need for further theoretical interpretation of the auxiliary signals.