3D Context Enhanced Region-based Convolutional Neural Network for End-to-End Lesion Detection

1 Introduction

• 3D context helps distinguish lesions from non-lesions, while existing CNN detection frameworks are typically designed for 2D images.The paper identifies a need for algorithms that incorporate 3D context [6].Source: §1 · p. 1
• 3D RPN methods face GPU-memory constraints and sparse 3D-box annotations, while 2D candidate-generation plus FPR methods are not end-to-end [6].The cited 3D RPN approach uses small patches and data augmentation [6].Source: §1 · pp. 1–2
• 3DCE aggregates neighboring-slice feature maps in a 2D regional CNN, enabling end-to-end training and inference with 2D-box annotations.It can use popular 2D CNN backbones and pretrained weights such as VGG-16.Source: §1 · p. 2
• Sensitivity at 4 false positives per image rose from 80.32% to 84.37% on DeepLesion after adding 3D context with 3DCE.The paper applies the unified detector to lesions of varied types.Source: §1 · p. 2

2 Method

• Feature-level fusion aggregates neighboring 2D feature maps to add 3D information while retaining the 2D backbone structure.This contrasts with data-level fusion, which may require pretrained weights to adapt substantially to 3D textures.Source: §2 · p. 2

2.1 3DCE

• R-FCN uses position-sensitive score maps and PSROI pooling to classify lesion proposals and regress their bounding boxes.The paper reports that R-FCN is faster, more accurate, and more memory-efficient than faster RCNN.Source: §2.1 · p. 3
• 3DCE shares Conv1–6 weights across slice images, concatenates their feature maps, and jointly optimizes RPN and improved R-FCN classification and box-regression losses.The authors set D = 10 and found M = 3–9 (18–54 mm) a reasonable context range; shared weights reduce parameters relative to 3D filters.Source: §2.1 · pp. 3–4

2.2 Implementation Details

• Implementation initialized Conv1–5 from ImageNet-pretrained VGG-16 and randomly initialized the remaining layers.Training used SGD with a base learning rate of 0.001, reduced by a factor of 10 after the fourth and fifth epochs.Source: §2.2 · p. 4

3.1 DeepLesion Dataset

• On DeepLesion, 3DCE used 32,735 lesions from 32,120 slices across 10,594 CT studies, split by patient into training, validation, and test sets.The dataset spans varied lesion types and sizes; volumes were resampled to 2 mm slice intervals.Source: §3.1 · pp. 4–5

3.2 Results and Discussion

• 3DCE achieved the best accuracy among compared methods, including across the original and all-lesion test sets and two overlap criteria.The comparison reports sensitivity at 4 false positives per image; the all-lesion set adds more extensive annotations.Source: §3.2 · pp. 5–6, Table 2 · p. 7
• All lesion types improved with 3DCE, bone lesions improved most, and lesions smaller than 10 mm benefited more from 3D context.Lung, mediastinum, and liver lesions had high sensitivity; finer slice intervals were associated with better performance in this split.Source: §3.2 · p. 6, Table 1 · p. 6
• Adding radiologist annotations on 300 slices increased ground-truth lesions from 305 to 768, and the all-lesion test set had lower sensitivity at 4 false positives per image.Some newly labeled lesions were too small or unmeasurable and absent from training, affecting performance.Source: §3.2 · p. 6
• 3DCE’s time complexity grows roughly linearly with input slices, though cached neighboring-slice feature maps can largely reduce the extra cost on volumetric data.A scratch-trained 3D CNN reached 79.7% sensitivity at 4 false positives per image, compared with 3DCE’s 84.4%.Source: §3.2 · p. 7

3.3 Results on the official data split of DeepLesion

• On the official patient-level split, overall result trends remained similar, except that slice interval was no longer a critical factor for 3DCE.The authors suggest split differences, removal of noisy samples, and random factors may explain changes.Source: §3.3 · pp. 7, 9

• 3DCE consistently improved lesion-detection accuracy on DeepLesion.Source: §4 · p. 10
• 3DCE is memory-friendly, end-to-end, and simple to implement and train.Source: §4 · p. 10
• The unified detector may serve as an initial screening tool whose results can be sent to specialist systems.Source: §4 · p. 10