PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Lesion Graphs in the Wild: Relationship Learning and Organization of Significant Radiology Image Findings in a Diverse Large-scale Lesion Database

Ke Yan, Xiaosong Wang, Le Lu, Ling Zhang, Adam Harrison, Mohammadhad Bagheri, Ronald Summers

arXiv:1711.10535v3cs.CV

TL;DR

The paper tackles the problem of organizing large collections of clinically annotated but semantically unsorted PACS lesions. It constructs DeepLesion and learns lesion embeddings from weak cues using a triplet network with sequential sampling. The resulting representations support lesion retrieval and intra-patient matching, while the framework remains bounded by assumptions about lesion detection and incomplete study coverage.

  • Problem

    Radiology PACS archives contain many clinically significant lesion annotations, but the lesions are largely unsorted and lack semantic labels such as type and location.

  • Method

    The paper mines DeepLesion from PACS and learns lesion embeddings with a triplet network using type, self-supervised body location, size, and sequential sampling.

  • Results

    The learned embeddings show promising results for content-based lesion retrieval and intra-patient lesion matching.

  • Takeaways & Limitations

    The framework can serve as a generic lesion search engine, classifier, and matching tool, with further specialist processing applied to retrieved or classified lesions.

  • Takeaways & Limitations

    The matching setting assumes lesions are detected or marked, while lesion visibility varies across studies because scan ranges and radiologist markings are incomplete.

Abstract

from arXiv · show

Radiologists in their daily work routinely find and annotate significant abnormalities on a large number of radiology images. Such abnormalities, or lesions, have collected over years and stored in hospitals' picture archiving and communication systems. However, they are basically unsorted and lack semantic annotations like type and location. In this paper, we aim to organize and explore them by learning a deep feature representation for each lesion. A large-scale and comprehensive dataset, DeepLesion, is introduced for this task. DeepLesion contains bounding boxes and size measurements of over 32K lesions. To model their similarity relationship, we leverage multiple supervision information including types, self-supervised location coordinates and sizes. They require little manual annotation effort but describe useful attributes of the lesions. Then, a triplet network is utilized to learn lesion embeddings with a sequential sampling strategy to depict their hierarchical similarity structure. Experiments show promising qualitative and quantitative results on lesion retrieval, clustering, and classification. The learned embeddings can be further employed to build a lesion graph for various clinically useful applications. We propose algorithms for intra-patient lesion matching and missing annotation mining. Experimental results validate their effectiveness.

1. Introduction

The paper addresses the challenge of exploiting clinically annotated but unsorted PACS lesion data by learning embeddings that organize lesions according to type, location, and size. It introduces DeepLesion and demonstrates applications in retrieval and intra-patient matching.

  • PACS archives contain extensive radiology findings, but their annotations are unsorted and lack semantic labels such as lesion type and location.
  • The framework learns lesion embeddings whose similarity reflects type, 3D body location, and size, using weak cues to reduce manual annotation.
  • The embeddings support content-based retrieval, including finding lesions with semantically similar body structures not specified by training labels.
  • The framework also supports intra-patient lesion matching, which can automate comparisons across follow-up CT studies.

2. Related work

Related work situates the paper at the intersection of deep metric learning and lesion management. Its contribution is to adapt similarity learning to diverse lesions using weak supervision rather than relying only on manually labeled lesion categories.

  • Deep Metric Learning: Deep metric learning learns representations that preserve desired similarity relationships between samples.
  • Deep Metric Learning: Triplet networks learn relative distances from an anchor, a same-label positive, and a different-label negative sample.
  • Deep Metric Learning: The paper’s sequential sampling strategy uses weak cues because the dataset lacks well-defined supervision, extending hierarchical similarity-learning ideas.
  • Lesion Management: Existing lesion-management methods commonly target one lesion type and learn similarity from manual annotations or radiology reports.

3. DeepLesion Dataset

DeepLesion is a large PACS-derived dataset of clinically significant CT findings, constructed by extracting radiologist-marked lesion diameters and converting them into padded bounding boxes. Its scale and extraction workflow support lesion-learning research from routine clinical data.

  • DeepLesion contains over 32K clinically significant findings mined from a major institute’s PACS.
  • The workflow is designed to scale to multiple institutional PACS and potentially provide massive datasets.
  • The dataset includes 32,120 axial CT slices from 10,594 studies and 4,427 patients, totaling 32,735 lesions.
  • Lesion diameters were extracted from radiologist annotations, converted to image-plane coordinates, and enclosed in boxes with 5-pixel padding.
  • Each lesion is represented using a cropped patch with 50 mm padding and three neighboring CT slices as a 3-channel image.

4. Learning Lesion Embeddings

The framework learns lesion embeddings from weak cues—type, relative body location, and size—using a hierarchical sampling and triplet-training strategy. A self-supervised body-part regressor supplies the relative z-coordinate, while sequential sampling organizes similarity across cues.

  • 4.1. Supervision Cues: The model uses lesion type, relative body location, and size as weak supervision cues because exhaustive PACS annotation is prohibitively time-consuming.Type labels are propagated from manually labeled seed samples; size measurements are radiologist-annotated.
  • 4.1. Supervision Cues: SSBR learns an appearance-based relative z-coordinate by ranking equidistant CT slices and matching score differences to physical distances.After convergence, slice scores are normalized to [0, 1] without requiring body-part labels.
  • 4.1. Supervision Cues: SSBR is trained on random axial crops and with hard volumes oversampled to address zoomed-in scans and rare body parts.Hard volumes are identified using the correlation between slice indices and predicted scores.
  • 4.2. Sequential Sampling: The conditional ranking scheme prioritizes lesion type, then location, then size, rather than treating all cues equally.Lesions with the same type are clustered first; location and size provide progressively finer ranking within that structure.
  • 4.2. Sequential Sampling: Sequential sampling forms lesion sequences A–E with progressively weaker similarity, decomposing each sequence into triplets ABC, ACD, and ADE.Similarity thresholds define cue agreement, while hard triplet mining is avoided because the cues are noisy.
  • 4.2. Sequential Sampling: Hierarchical margins satisfy m3 > m2 > m1 > 0, placing dissimilar types farther apart than dissimilar locations, followed by size.The triplet network uses a VGG-16 backbone and multi-scale lesion features to learn the embedding space.

5. Lesion Organization

After learning lesion embeddings, the framework constructs a lesion graph for retrieval and intra-patient lesion matching. Matching addresses cross-study variation and groups lesions that represent the same instance within each patient.

  • 5. Lesion Organization: The learned lesion graph supports content-based retrieval through nearest-neighbor search and requires additional techniques for intra-patient matching.The graph directly addresses retrieval, whereas matching must account for lesion identity across studies.
  • 5. Lesion Organization: Intra-patient matching assumes lesions are already detected or marked by radiologists in every study.The setting includes appearance changes, organ movement, contrast-phase differences, variable scan ranges, and multiple series.
  • 5. Lesion Organization: The matching algorithm builds an intra-patient lesion graph and removes edges connecting different lesion instances using embedding distances.It first merges sufficiently close lesions from different series within a study, then compares lesions across all studies of the patient.

6. Experiments

Experiments evaluate whether lesion embeddings organize clinically relevant similarity and support retrieval, classification, clustering, and intra-patient matching. Results show organization by type, location, size, and appearance, with strong matching performance and benefits from combining supervision cues and multi-scale features.

  • Content-based lesion retrieval: The learned embeddings form clusters organized by lesion type and location, with samples also roughly similar in size.Lung, kidney, and bone lesions split into anatomically meaningful subclusters, while abdomen, liver, and kidney lesions remain close in feature space.
  • Content-based lesion retrieval: Retrieved lesions are often the same patient’s lesion instances, while cross-patient retrieval finds semantically similar structures despite coarse supervision labels.Examples include spiculated lung nodules, paraaortic lymph nodes, and pancreatic-tail lesions; some failure cases remain.
  • Quantitative evaluation: Adding location and size supervision produces the best lesion-type retrieval, outperforming training with lesion type alone.The authors interpret these cues as supplementary information that can make embeddings more organized and act as regularizers.
  • Quantitative evaluation: Fine-level features significantly improve type and size prediction, whereas coarse-scale features slightly improve location prediction.The ablation results support combining visual features from different context levels.
  • Intra-patient lesion matching: The proposed embedding achieves the highest AUC for intra-patient lesion matching without matched-lesion training annotations, registration, or organ masks.Evaluation groups 1,313 lesions from 103 patients into 593 manually defined lesion-instance groups.

7. Conclusion and Future Work

The paper presents DeepLesion and a triplet-network framework for organizing PACS-derived lesions by type, location, and size. The resulting embeddings support retrieval, classification, and intra-patient matching, with future work aimed at richer semantic information and a lesion knowledge graph.

  • Conclusion: DeepLesion contains significant radiology findings mined from PACS, while lesion embeddings model similarity in type, location, and size.The framework requires manual class labels only for some seed images.
  • Conclusion: The framework produces promising results for content-based lesion retrieval and intra-patient lesion matching.The authors describe it as a generic lesion search engine, classifier, and matching tool.
  • Future work: Future work will add finer-grained semantic information from radiology reports, specialized datasets, or active learning to build a lesion knowledge graph.

8. Supplementary Material

The supplementary material visualizes DeepLesion, details self-supervised body-part localization, and illustrates retrieval and intra-patient lesion matching. These analyses support the validity of the location representation and the practical use of lesion graphs for matching.

  • DeepLesion Dataset: DeepLesion exhibits considerable lesion diversity, with overlapping locations among neighboring anatomical categories and widespread distributions for bone and soft-tissue lesions.The scatter map shows that relative body locations are consistent with physical positions, while inter-subject variability creates overlap between neighboring types.
  • Location and Size: Lesion coordinates combine axial-slice body masks and relative lesion-center positions with z-coordinates predicted by the self-supervised body-part regressor.The axial x- and y-coordinates are normalized to 0–1, while the regressor supplies the slice-level z-coordinate.
  • Self-Supervised Body-Part Regressor: 95.99% classification accuracy was achieved for chest, abdomen, and pelvis slice classes using SSBR, with errors concentrated in ambiguous transition regions.SSBR was trained on unlabeled CT volumes, requiring zero annotation effort for its training data.
  • Content-Based Lesion Retrieval: Most retrieved lesions match queries in type, location, and size, and often come from semantically similar body structures not specified by training labels.The supplementary retrieval examples also include failure cases in which dissimilar types are retrieved.
  • Intra-Patient Lesion Matching: The lesion graph and matching algorithm can accurately match lesions across multiple studies for a sample patient.The supplementary figures visualize the patient’s lesions, lesion graph, and extracted lesion sequences.
Loading 1711.10535v3…