PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to detect chest radiographs containing lung nodules using visual attention networks

Emanuele Pesce, Petros-Pavlos Ypsilantis, Samuel Withey, Robert Bakewell, Vicky Goh, Giovanni Montana

arXiv:1712.00996v3stat.MLcs.CV

TL;DR

Small lung nodules are difficult to detect on chest radiographs, and deep-learning detection commonly requires manual annotations that are hard to obtain at scale. This paper combines report-derived weak labels with lesion annotations in two visual-attention architectures. The authors report stronger classification and localization performance for CONAF against other methods, while RAMAF improves on the original approach when annotated images are available.

  • Problem

    Detecting small pulmonary nodules on chest radiographs is challenging, while deep-network training typically requires large numbers of manually annotated images that are difficult to obtain.

    Source: §1 · p. 2

  • Method

    The authors train two visual-attention classifiers using archive-scale weak labels and, where available, manually delineated lesion boxes to provide attention feedback.

    Source: §1 · p. 3

  • Results

    CONAF outperforms other state-of-the-art methods, and RAMAF improves on the original approach when annotated images are available.

    Source: §5 · p. 16

  • Takeaways & Limitations

    Combining many reported images with a high-quality subset of annotated lesions improves sensitivity while retaining an acceptably low level of false positives, which the authors identify as essential for clinical use.

    Source: §5 · p. 17

    RAMAF uses a fixed number of glimpses; future work could let it adapt the number and size of glimpses to each image.

    Source: §5 · p. 17

Abstract

from arXiv · show

Machine learning approaches hold great potential for the automated detection of lung nodules in chest radiographs, but training the algorithms requires vary large amounts of manually annotated images, which are difficult to obtain. Weak labels indicating whether a radiograph is likely to contain pulmonary nodules are typically easier to obtain at scale by parsing historical free-text radiological reports associated to the radiographs. Using a repositotory of over 700,000 chest radiographs, in this study we demonstrate that promising nodule detection performance can be achieved using weak labels through convolutional neural networks for radiograph classification. We propose two network architectures for the classification of images likely to contain pulmonary nodules using both weak labels and manually-delineated bounding boxes, when these are available. Annotated nodules are used at training time to deliver a visual attention mechanism informing the model about its localisation performance. The first architecture extracts saliency maps from high-level convolutional layers and compares the estimated position of a nodule against the ground truth, when this is available. A corresponding localisation error is then back-propagated along with the softmax classification error. The second approach consists of a recurrent attention model that learns to observe a short sequence of smaller image portions through reinforcement learning. When a nodule annotation is available at training time, the reward function is modified accordingly so that exploring portions of the radiographs away from a nodule incurs a larger penalty. Our empirical results demonstrate the potential advantages of these architectures in comparison to competing methodologies.

1 Introduction

Small pulmonary nodules are difficult to detect on chest radiographs, while training deep networks traditionally depends on annotations that are costly to scale. The paper investigates combining archive-scale weak labels with lesion bounding boxes through visual-attention feedback, and reports improved classification performance.

1 Introduction

  • Plain-film nodule detection is challenging because small or ill-defined lesions can be obscured by overlapping thoracic structures, and large manually annotated image sets are scarce.The cited literature reports sensitivity rates of 51%–71% for lesions larger than 5 mm.Source: §1 · p. 2
  • CONAF uses saliency maps and annotated lesion locations to penalize localization error alongside classification error during training.The localization loss is back-propagated to provide visual feedback on the network’s learned features.Source: §1 · p. 3
  • RAMAF processes sequential image portions and rewards attention that overlaps annotated lesions, with the authors reporting faster convergence and increased classification performance.Its attention policy selects locations using information accumulated at earlier steps.Source: §1 · p. 4
  • The study tests whether a large weakly labelled archive combined with a smaller set of manually annotated lesions can improve chest-radiograph classification.The authors collected over 700,000 historical radiographs and assessed the classifier on an independent, manually curated dataset of over 6,000 exams.Source: §1 · pp. 3, 5
  • The authors report that combining a large set of weakly annotated examples with a relatively small annotated subset can drastically enhance classification performance.Figure 1 shows three successful CONAF detections, with probability heatmaps alongside radiologist-drawn bounding boxes.Source: Figure 1 · p. 4, §1 · p. 5

2 A repository of chest radiographs

The study assembled historical chest radiographs and reports, assigned report-derived labels, and manually delineated lesions in a subset. The collection includes non-pulmonary abnormalities as well as suspected lesions, and lesion size is represented by the bounding box’s longest side.

2 A repository of chest radiographs

  • Of 745,479 archived exams, 430,067 had both images and reports after pediatric exams were removed; images were resized to 448×448 pixels.The exams came from Guy’s and St. Thomas’ NHS Foundation Trust and covered January 2005 to March 2016.Source: §2 · p. 5
  • An NLP system assigned mutually exclusive normal, lesions, and other-abnormality labels from radiology reports, identifying 101,766 normal, 23,132 lesion, and 305,169 other-abnormality exams.The system extended an existing NLP system for extracting clinical findings from radiology reports.Source: §2 · p. 5
  • For this study, “lesion” includes both nodules under 3 cm and masses at least 3 cm, and training and testing used radiographs reporting possible lesions rather than only CT- or pathology-confirmed cancers.Lesions may overlap ribs, the mediastinum, diaphragm, or heart on radiographs.Source: §2 · p. 5
  • An experienced radiologist manually annotated 2,196 images, drawing bounding boxes around 3,253 lesions; lesion size was approximated by the longest bounding-box side in millimeters, an upper bound on real nodule size.Figure 2 shows the frequency distribution of lesion size across all annotated images.Source: §2 · p. 6, Figure 2 · p. 6

3 Proposed architectures

The proposed architectures use lesion annotations alongside weak labels to improve image classification through visual attention feedback. CONAF compares saliency maps with lesion annotations, while RAMAF learns where to look through sequential glimpses and annotation-based rewards.

3.1 Convolution networks with attention feedback (CONAF)

  • The training set contains 430,067 weakly labelled radiographs, including 2,196 images with lesion bounding-box annotations.Labels distinguish normal exams, lesion-containing exams, and exams reporting other abnormalities.Source: §3.1 · p. 7
  • CONAF uses a shared feature extractor for classification and localisation, with a scoremap branch estimating lesion positions and a classifier predicting image labels.The localisation scoremap is compared with a bounding-box-derived target for annotated images.Source: §3.1 · pp. 7–8
  • The CONAF training objective combines classification and localisation losses, with localisation feedback available only for images carrying bounding boxes.Its localisation target smooths the rectangular annotation into a Gaussian-shaped high-probability region centred on the lesion.Source: Figure 3 · p. 8, §3.1 · pp. 8–9

3.2 Recurrent attention model with annotation feedback (RAMAF)

  • RAMAF processes an image through sequential glimpses, using information from prior steps to choose where to look next.Each glimpse contains two differently sized patches centred at the same location.Source: §3.2 · pp. 9–10
  • RAMAF combines classification feedback with an annotation-based spatial reward that rewards glimpse centres falling inside lesion bounding boxes.The glimpse encoder and recurrent Core RAM summarize observations to guide the next location choice.Source: §3.2 · pp. 10–11

4 Experimental results

Across the classification and localisation experiments, CONAF performs strongly against competing methods, while RAMAF improves on RAM when using annotated images. The experiments also specify implementation choices and comparator architectures.

4.1 Further implementation details

  • CONAF training alternated weak-label and bounding-box batches, sampled with probabilities 0.8 and 0.2, respectively, to limit localisation overfitting.The loss weights were λ1 = 10 and λ2 = 0.1; the authors selected these on validation performance.Source: §4.1 · p. 11
  • RAMAF used seven glimpses per image, each combining a 70 × 70 high-resolution window and a 140 × 140 low-resolution window.Training used back-propagation through time with Adam.Source: §4.1 · p. 11

4.2 Competing architectures

  • The comparison included weakly supervised classification/localisation methods and fully supervised object detectors, including OverFeat and an CNN–LSTM bounding-box predictor.The weakly supervised comparators were Oquab et al. (2015) and Zhou et al. (2016).Source: §4.2 · p. 12

4.3 Lesion classification performance

  • CONAF outperforms other methods on average accuracy, F1-measure, and sensitivity; Class Activation Maps achieve the highest precision for lesion detection versus Others.The comparison uses a test set of 6,131 images whose weak labels were manually validated by two radiologists.Source: Table 2 · p. 12, §4.3 · p. 13

4.4 Lesion localisation performance

  • CONAF localisation was evaluated by thresholding scoremaps from 0.2 to 0.8 and counting candidate boxes with at least 25% overlap with ground truth as true positives.Precision and sensitivity were reported; accuracy and F1 could not be calculated because the task had no true negatives.Source: §4.4 · p. 14
  • CONAF achieves superior sensitivity and average overlap in localisation, while OverFeat achieves the best precision.CONAF also has greater sensitivity than all other methods; its sensitivity/precision falls between weakly supervised and object-detection methods.Source: Table 3 · p. 13, §4.4 · p. 14
  • RAMAF detected 82% of test-set bounding boxes, compared with 55% for RAM, and learned approximately five times faster.Comparable localisation metrics could not be obtained for RAM/RAMAF; box detection was measured by overlap with at least one glimpse.Source: §4.4 · p. 14

5 Discussion and conclusions

The authors report improved image-classification performance using F1 and favorable CONAF localization results, while presenting soft- and hard-attention approaches that use lesion annotations. They note that comparisons across studies may be misleading and describe future scope for improving RAMAF.

5 Discussion and conclusions

  • The study uses a large historical chest-radiograph database from two UK hospitals, with images from over forty scanners and a diversified adult population.The authors describe it as the largest study to date exploring deep learning for pulmonary lesion detection, to their knowledge.Source: §5 · pp. 15–16
  • CONAF combines a saliency-map localization loss with classification error to refine attention using lesion-position feedback.Its soft-attention approach uses high-level features to focus on salient image regions.Source: §5 · p. 16
  • RAMAF uses recurrent hard attention and spatial feedback rewards to explore image regions, while stochastic region selection requires reinforcement learning rather than back-propagation.Hard-attention computation is independent of input-image size, unlike soft attention, whose complexity scales with image pixels.Source: §5 · pp. 16–17
  • Image-classification results improve over documented methodologies by F1, while combining reported images with annotated lesions improves sensitivity at an acceptably low false-positive level.For localization, CONAF has higher sensitivity than other algorithms, while OverFeat and Stewart’s method achieve higher precision at much lower sensitivity; CONAF also achieves very high overlap with ground-truth regions.Source: §5 · p. 17
  • Direct comparisons with published results may mislead because studies differ in which exam types they compare, and this study’s image labels are inevitably noisy.The authors describe their empirical results as promising despite label noise.Source: §5 · p. 17
  • RAMAF could be extended to adapt the size and number of glimpses to the context needed to classify each image.The described approach currently uses a fixed number of glimpses.Source: §5 · p. 17

A The NLP system for automated image tagging

The NLP system labels radiographs from radiological reports as lesion or no reported lesion. The authors assume most labels are accurate despite occasional reporting, human, or NLP errors.

A The NLP system for automated image tagging

A.1 Entity detection

Entity detection maps report phrases to medical concepts using RadLex and MeSH. The system also uses matching procedures to handle alternate forms and misspellings.

A.1 Entity detection

  • The system identifies medical entities using RadLex and MeSH, hierarchically structured radiological and general medical lexicons.Source: §A.1 · p. 24
  • Entities are assigned to one of four semantic classes: Clinical Finding, Body Location, Descriptor, or Medical Device.These classes distinguish abnormalities, anatomical areas, descriptive adjectives, and devices seen on radiographs.Source: §A.1 · p. 24
  • After tokenizing, lowercasing, and lemmatizing report sentences, the system matches the longest phrase to a RadLex or MeSH concept, with Wikipedia redirects as a fallback.SimString cosine similarity matching uses a threshold of 0.85 to accommodate misspellings.Source: §A.1 · p. 24

A.2 Negation detection

Negation detection combines NegEx predictions with a reduced grammatical-relations graph to classify each identified entity as negated or affirmed.

A.2 Negation detection

  • The system obtains NegEx predictions, builds a Universal Dependencies relation graph, and retains only negation and or-disjunction relations.Source: §A.2 · p. 25
  • An entity is classified as negated when its words are negated or linked by or-disjunction to a negated word, or when NegEx marks it negated and it is the closest entity to the trigger without a sentence-level negation relation.Source: §A.2 · p. 25
  • If neither negation condition holds, the entity is classified as affirmed; unlike this approach, DEEPEN considers all first-order dependency relations between trigger and target.Source: §A.2 · p. 25

A.3 Relation classification

The relation classifier identifies directed “located in” and “described by” relations between entities in the same sentence, subject to semantic-class constraints. Separate CNN binary classifiers predict candidate relations using token embeddings and position features.

A.3 Relation classification

  • Relations are restricted to entities in the same sentence, with allowed argument classes differing between “located in” and “described by.”“Located in” links a Clinical Finding or Medical Device to a Body Location; “described by” links a Clinical Finding, Medical Device, or Body Location to a Descriptor.Source: §A.3 · p. 25
  • Separate CNN binary classifiers evaluate candidate relations using ordered token embeddings and features for each token’s distance from the relation arguments.The architecture applies two convolutional and max-pooling layers, followed by two fully connected layers and softmax classification.Source: §A.3 · p. 25
  • The relation datasets contain 1,100 “located in” examples and 1,100 “described by” examples, with candidate arguments more than 16 words apart labeled false.The “located in” set has 729 true and 371 false relations; the “described by” set has 507 true and 593 false relations.Source: §A.3 · p. 26
  • The report examples show NLP-assigned classes Normal, Lesion, and Others, with pleural effusion and consolidation placed in Others.The examples illustrate report classifications, not relation-classifier performance.Source: Figure A.1 · p. 27

A.4 Classification

The final NLP stage assigns report classes by matching extracted entities, negation attributes, and relations against radiologist-designed rules; reports matching no rule remain unclassified.

A.4 Classification

  • The system processes each report’s extracted entities, negation attributes, and relations against a set of 826 rules designed with expert radiologists.A matching rule assigns its corresponding radiological class.Source: §A.4 · p. 28
  • Reports that match no rule are left unclassified.Source: §A.4 · p. 28
  • Each rule maps to one of the radiological classes.Source: §A.4 · p. 28

A.5 Validation study

The validation study compared NLP report labels with independent labels from two radiologists on a random subset of reports. The reported results describe performance overall and across diseases grouped as Others.

A.5 Validation study

  • 4,652 randomly selected reports were independently labeled by two radiologists blinded to the images; about 7% were Lesion, 15% Normal, and the remainder Others.Source: §A.5 · p. 28
  • The NLP system achieved good performance overall, particularly for Normal exams.Table A1 reports average accuracy, F1, sensitivity, specificity, precision, and negative predictive value.Source: §A.5 · p. 28, Table A1 · p. 28
  • For diseases in Others, Table A2 reports precision, sensitivity, specificity, F1, and negative predictive value, alongside disease prevalence.The text describes NLP performance across the available diseases as very good but supplies no individual metric values here.Source: §A.5 · p. 28, Table A2 · p. 29

B The RAMAF model

RAMAF uses a recurrent policy with internal memory to choose sequential image glimpses and maximize expected reward. Its spatial reward accelerates learning relative to RAM, while the training objective also uses image labels and a learned reward baseline.

B The RAMAF model

  • The policy uses an LSTM hidden state to summarize prior glimpses and select actions, inducing a distribution over image-observation sequences whose expected reward is maximized.The paper frames this as a reinforcement-learning problem in a partially observable Markov decision process.Source: B The RAMAF model · p. 29
  • Because exact expectation is difficult under unknown environment dynamics, the model approximates its gradient with REINFORCE and computes recurrent-network gradients by backpropagation.Source: B The RAMAF model · p. 29
  • A constant baseline reward is included in the gradient estimate to reduce variance; the learned baseline is the expected cumulative reward under the policy.The algorithm raises action log-probability for above-baseline returns and lowers it for below-baseline returns.Source: B The RAMAF model · p. 30
  • RAMAF learns approximately five times faster than RAM, with the spatial reward directing attention toward regions likely to contain lesions.RAM lacks spatial reward and initially spends more time exploring irrelevant image portions.Source: B The RAMAF model · p. 30, Figure B.1 · p. 31
  • Accuracy is reported by lesion-size decile for both experiments, from 0.1 for the smallest nodules to 1.0 for the largest masses.Source: Figure B.2 · p. 31
Loading 1712.00996v3…