Detecting hip fractures with radiologist-level performance using deep neural networks

• One-year all-cause mortality after hip fracture exceeds 20%, and fewer than 50% of patients regain independent living; lifetime risk is 17.5% for women and 6% for men.Source: §1 · p. 1
• Although pelvic x-rays usually establish fracture diagnoses, 5–10% of at-risk patients undergo further imaging, and only a third of those studies demonstrate a fracture.Source: §1 · p. 1
• The study investigates convolutional neural networks for fracture detection and presents a large-scale study reporting human-level performance on this radiological task.Source: §1 · p. 2

2.1 Developing ground truth labels for hip fractures

• The authors describe hip fractures as having near-perfect ground-truth labels because cases are expected to appear in hospital records, with estimated label accuracy above 99%.Source: §2.1 · p. 2

2.2 Obtaining and efficiently labelling our dataset

• Combining surgical records and radiology reports initially yielded about 95% label accuracy; iterative model-guided review raised it to over 99.9% after expert labeling of 7.4% of the dataset.Source: §2.2 · p. 2
• The study included 53,278 images split by patient into training, validation, and held-out test sets; the test set comprised emergency-department referrals with 19% fracture prevalence.Source: §2.2 · p. 2

2.3 Managing medical data heterogeneity

• Preprocessing networks identify frontal pelvic films, localize the femoral neck, and exclude cases with implanted metal; neck localization reduces input images to 1024 × 1024 pixels while maintaining resolution.Source: §2.3 · pp. 2–3, Table 1 · p. 3

3.1 Model selection

• The pelvic x-rays were analyzed with DenseNet, an architecture with extensive feed-forward connections between layers.Source: §3.1 · p. 3
• The validation set guided selection of layer width, activation, loss functions, augmentation, regularization, and learning rate.Source: §3.1 · p. 3
• Each data-augmentation technique improved validation AUC by around 0.01, with translations, rotations, shears, and histogram matching used to address image variation.Source: §3.1 · p. 3
• DenseNet validation performance exceeded both a ten-layer fully convolutional CNN and pretrained networks by an absolute AUC improvement of 0.035 in each comparison.Source: §3.1 · p. 3

3.2 Evaluation

• Evaluation used a held-out test set of 3,354 images, including 348 fractures, and compared the algorithm with published automated work and radiology reports.Source: §3.2 · p. 4

3.3 Estimating human performance

• Radiology-report estimates of human performance are confounded by unreported information, while the study's upper-bound approach assumes no such information informed reports.Source: §3.3 · p. 4

• AUC 0.994: the model’s ROC curve is shown on hold-out test data, alongside an optimistic upper-bound point for human performance.The figure caption identifies the point as the optimistic upper bound of human performance.Source: Figure 1 · p. 4, §4 · p. 4
• Table 2 compares recent automated hip-fracture detection results; Kazi et al.’s upper-bound model required human neck-of-femur localization, while this system localized it automatically.Source: Table 2 · p. 5
• No significant difference was found between the model and the estimated human upper bound at either operating point on this dataset.The estimate uses original clinical radiology reports and assumes radiologists relied only on frontal pelvic x-rays, which is unlikely.Source: Table 3 · p. 5
• The comparison with Kazi et al. used a 50% fracture prevalence to make performance metrics comparable, because precision varies with prevalence.Kazi et al. reported results at a single operating point for each model.Source: §4 · p. 5
• The larger test-set analysis included 348 fractures and 2,997 non-fractures and compared model performance with original radiology reports.Radiologists often had additional information, including lateral x-rays, clinical information, and follow-up studies; the model was evaluated at high-precision and high-recall operating points.Source: §4 · p. 5

• The model achieved human-level hip-fracture detection performance in a large-scale study and state-of-the-art performance by a large margin over recent work.The authors report an AUC of 0.994, which they say is the highest then reported for automated diagnosis in a large-scale medical task, to their knowledge.Source: §5 · p. 5
• Iterative labeling improved label accuracy from around 95% to over 99.9% while experts hand-labeled only 7.4% of the dataset.Experts reviewed false positives and false negatives from partially trained networks rather than the entire dataset.Source: §5 · p. 5
• The authors report that AUC 0.994 was achievable without pre-training and that very deep networks were useful in this setting.Source: §5 · p. 6
• Estimating baseline human performance is difficult because diagnostic decisions can use unreported external information and radiologists use inexact language.The authors describe their human upper-bound estimate as optimistic and say they are working to test the system against experts directly in a controlled environment.Source: §5 · p. 6