PaperlayerView exact revision on arXiv

Source-linked AI summary

TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays

Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Ronald M. Summers

arXiv:1801.04334v1cs.CV

TL;DR

The paper tackles scarce machine-learnable medical imaging data and the challenge of modeling radiologists’ reasoning by exploiting free-text radiological reports. It introduces TieNet, a multi-attention text-image embedding network for chest X-ray annotation, classification, and reporting, achieving improved classification performance while generated-report quality remains limited.

  • Problem

    Medical imaging has few large-scale machine-learnable datasets, and automated systems lack techniques that mimic radiologists’ holistic reasoning over multiple findings.

  • Method

    TieNet integrates report-derived text embeddings and image features through multi-level attention in an end-to-end CNN-RNN framework for auto-annotation, classification, and preliminary report generation.

  • Results

    6% increase on average in AUC for all disease categories was achieved over the state-of-the-art on the unseen, hand-labeled OpenI dataset.

  • Takeaways & Limitations

    Paired radiological reports can provide prior knowledge that improves chest X-ray disease classification while supporting a joint classification-and-reporting system.

  • Takeaways & Limitations

    Generated reports can read well while describing diseases absent from images or missing diseases that are present.

Abstract

from arXiv · show

Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due to (1) shortage of large-scale machine-learnable medical image datasets, and (2) lack of techniques that can mimic the high-level reasoning of human radiologists that requires years of knowledge accumulation and professional training. In this paper, we show the clinical free-text radiological reports can be utilized as a priori knowledge for tackling these two key problems. We propose a novel Text-Image Embedding network (TieNet) for extracting the distinctive image and text representations. Multi-level attention models are integrated into an end-to-end trainable CNN-RNN architecture for highlighting the meaningful text words and image regions. We first apply TieNet to classify the chest X-rays by using both image features and text embeddings extracted from associated reports. The proposed auto-annotation framework achieves high accuracy (over 0.9 on average in AUCs) in assigning disease labels for our hand-label evaluation dataset. Furthermore, we transform the TieNet into a chest X-ray reporting system. It simulates the reporting process and can output disease classification and a preliminary report together. The classification results are significantly improved (6% increase on average in AUCs) compared to the state-of-the-art baseline on an unseen and hand-labeled dataset (OpenI).

1. Introduction

The paper addresses limited machine-learnable medical image data and the difficulty of modeling radiologists’ holistic reasoning by using free-text reports as prior knowledge. TieNet jointly supports chest X-ray auto-annotation, classification, and preliminary reporting.

  • Medical imaging lacks large-scale publicly available datasets, while annotation requires professional training and domain knowledge.
  • Radiologists integrate multiple correlated findings into holistic clinical interpretations, motivating multi-disease diagnostic and reporting systems.
  • TieNet combines image features with report-derived text embeddings in an end-to-end multi-task CNN-RNN architecture with multi-level attention over words and image regions.
  • The framework uses paired image-report data for auto-annotation and can generate disease classifications and preliminary reports from images.
  • 6% increase on average in AUC for all disease categories was achieved over the state-of-the-art on the unseen, hand-labeled OpenI dataset.

2. Related work

Prior work established large-scale chest X-ray datasets from paired images and reports but largely reduced reports to binary disease labels. TieNet extends this direction by learning richer text-image embeddings with multi-attention for classification and report generation.

  • Existing CADe methods typically target one particular disease or lesion rather than multiple disease types.
  • A prominent chest X-ray dataset contains 112,120 front-view images from 30,805 unique patients, extracted from paired images and radiological reports.
  • Radiological reports contain richer information than binary disease labels, including disease location and severity.
  • TieNet differs from related visual captioning and attention approaches by combining multi-attention models with image and text features.

3. Text-Image Embedding Network

TieNet jointly learns text and image representations from radiological reports and chest X-rays through an end-to-end CNN-RNN architecture with multi-level attention. These representations support multi-label classification, report generation, and text-guided visual pooling.

  • Report information: Radiological reports encode clinical findings, impressions, negation, uncertainty, location, and severity beyond disease keywords.This richer information motivates learning report embeddings rather than using only mined disease labels.
  • End-to-end CNN-RNN: TieNet processes an image and a word sequence with a CNN-RNN model, using ImageNet-pretrained convolutional layers and an LSTM decoder.The CNN produces spatial activations, while the LSTM generates the report one word at a time from visual and textual context.
  • Attention-encoded text embedding: Attention-encoded text embedding combines salient LSTM hidden states into multiple global representations, then applies max-over-r pooling to obtain a text embedding.The attention weights also support visualization of which hidden states contribute to each embedded representation.
  • Saliency-weighted visual embedding: Saliency-weighted global average pooling reuses text attention to weight word-specific spatial attention maps and produce a global visual embedding.The resulting map highlights spatial regions of the convolutional features with more meaningful information.
  • Joint classification: The model concatenates global text and visual embeddings for multi-label classification using 15 disease or no-finding classes.Class-weighted training addresses severe differences in the numbers of examples across disease categories.
  • Medical image auto-annotation: For auto-annotation, TieNet uses paired image-report inputs and back-propagates classification loss while balancing contributions from report and image features.The report can provide easier-to-learn features, so training controls the influence of these paired modalities.

4. Dataset

The experiments use ChestX-ray14 for common thorax disease classification and OpenI for evaluation on radiographs collected from multiple institutes. ChestX-ray14 includes NLP-mined training labels and a hand-labeled subset for evaluation.

  • ChestX-ray14: ChestX-ray14 contains 14 common thorax disease labels for chest X-ray classification and localization.The categories include findings such as atelectasis, cardiomegaly, effusion, pneumonia, pneumothorax, edema, and hernia.
  • Hand-labeled evaluation: The study hand-labeled 900 testing-set reports with two radiologists to evaluate the 14 finding categories.A 30-report trial set was used to synchronize annotation criteria before independent annotation.
  • OpenI: OpenI provides 3,851 unique radiology reports and 7,784 associated frontal and lateral images collected from multiple institutes.The experiments use the same 14 finding categories and 3,643 unique front-view images with corresponding reports.

5. Experiments

The experiments evaluate TieNet for image classification, auto-annotation, and report generation using report, image-report, image, and image-generative-report inputs. TieNet improves classification across datasets and produces higher-scoring reports than the baseline, while examples reveal missed and false predictions.

  • Auto-annotation of images: Over 0.90 #wAVG AUC is achieved with image-report pairs on ChestX-ray14 and the Hand-labeled set.Report-only inputs exceed 0.87 AUC on these datasets.
  • Classification and reporting evaluation: 2.3%–5.7% #wAVG AUC improvement is achieved by TieNet over the image-only baseline across all disease categories and three datasets.The gain is larger on the Hand-labeled and OpenI sets than on ChestX-ray14.
  • Classification and reporting evaluation: Higher BLEU, METEOR, and ROUGE-L scores are obtained than with the baseline image-captioning model.These metrics compare generated reports with original reports using word-overlap statistics.
  • Classification and reporting evaluation: The experiments compare report-only, image-plus-report, image-only, and image-plus-generative-report inputs across ChestX-ray14, Hand-labeled, and OpenI datasets.ROC curves and AUC values are used for classification evaluation.
  • Qualitative results: Qualitative examples show missed findings, erroneous highlighted words, and false predictions that a radiologist could observe in the image.The examples include missed Effusion and Mass findings, as well as false Mass and Consolidation predictions.

6. Conclusion

The paper presents TieNet as an end-to-end CNN-RNN network that learns joint image and text representations from retrospective radiological data. It reports improved multilabel classification but identifies generated-report quality as an area requiring further improvement.

  • Conclusion: TieNet integrates multi-level attention into an end-to-end CNN-RNN framework for learning distinctive image and text representations.The framework is trained jointly for auto-annotation and reporting tasks.
  • Conclusion: Significant improvements are achieved in multilabel disease classification, while generated-report quality still has substantial room for improvement.Future work targets disease words, their attributes, and correlations between image findings and generated descriptions.

A. More Experiment Results

Additional qualitative experiments provide more classification and reporting examples, pairing images and predictions with original and generated reports. Text-attention intensity and prediction correctness are visually encoded.

  • A. More Experiment Results: Twenty additional cases extend the four main-paper examples with images, classification predictions, original reports, generated reports, and text attention.The appendix labels these cases E–X and uses varying saturation levels for text attention.
  • A. More Experiment Results: Green marks correct predictions, red marks false predictions, and blue marks missed predictions.Darker red indicates higher text-attention weights.
Loading 1801.04334v1…