Source-linked AI summary
Towards Automatic Threat Detection: A Survey of Advances of Deep Learning within X-ray Security Imaging
Samet Akcay, Toby Breckon
TL;DR
X-ray screening is important but human performance is affected by exhaustion, clutter, and concealed objects, motivating automated methods. This survey taxonomises conventional and deep learning approaches, reviews datasets and challenges, and concludes that limited, imbalanced data constrain generalisable real-time deployment.
Problem
Human operators face reduced detection performance and decision speed in cluttered X-ray screening, while automated screening remains understudied because of limited data and advanced learning requirements.
Method
The paper surveys approximately 130 selected works, taxonomising conventional machine learning and deep learning across classification, detection, segmentation, and anomaly detection, while reviewing datasets and evaluation criteria.
Results
The review covers the strengths and weaknesses of current techniques and provides a structured account of datasets, algorithms, open challenges, and future directions.
Takeaways & Limitations
Large, well-balanced datasets and domain adaptation are identified as directions for improving algorithm generalisation across X-ray machines and toward real-time deployment.
Takeaways & Limitations
The lack of large, well-balanced datasets limits the design of deep learning algorithms generalisable enough for real-time deployment.
Abstract
from arXiv · showhide
X-ray security screening is widely used to maintain aviation/transport security, and its significance poses a particular interest in automated screening systems. This paper aims to review computerised X-ray security imaging algorithms by taxonomising the field into conventional machine learning and contemporary deep learning applications. The first part briefly discusses the classical machine learning approaches utilised within X-ray security imaging, while the latter part thoroughly investigates the use of modern deep learning algorithms. The proposed taxonomy sub-categorises the use of deep learning approaches into supervised, semi-supervised and unsupervised learning, with a particular focus on object classification, detection, segmentation and anomaly detection tasks. The paper further explores well-established X-ray datasets and provides a performance benchmark. Based on the current and future trends in deep learning, the paper finally presents a discussion and future directions for X-ray security imagery.
1. Introduction
X-ray screening remains important but is affected by human and data limitations, motivating automated detection. This survey reviews conventional and deep-learning methods, datasets, and open challenges.
- Manual screening is affected by operator exhaustion, job satisfaction, cluttered bags, and concealed objects such as laptops.These factors reduce detection performance or decision speed.
- Automated object detection is motivated by the need to improve operator alertness and detection and response time.
- The field has used conventional image analysis and machine learning for classification, detection, and segmentation, while deep learning has more recently been adopted.CNN-based methods are described as outperforming conventional machine learning methods.
- The survey identifies approximately 213 relevant articles, including 36 using deep learning, and narrows its review scope to 130 papers.
- Its contributions include a taxonomy of classical and deep learning methods, an overview of large training datasets, and discussion of open problems and future directions.
- The paper organizes the literature around computer vision tasks and reviews datasets and evaluation criteria before discussing conventional and deep learning approaches.
2. Background: X-ray Imaging
X-ray imaging forms images by measuring attenuation as beams pass through objects. Modern systems extend this principle with multiple energies and viewpoints to characterize materials and object structure.
- An X-ray tube emits beams that penetrate scanned objects, whose material density attenuates the X-ray signal.
- The imaging formulation relates measured intensity to initial intensity, material thickness, and the linear attenuation coefficient.Higher-density materials produce greater attenuation and lower measured intensity.
- Multi-energy machines produce images at different energies to estimate object density and effective atomic number, then convert them into pseudo-coloured images.
- State-of-the-art machines also acquire scans from multiple viewpoints to inspect objects from different angles.
3. Datasets
The survey reviews X-ray security datasets used in the literature, including collections with varied sizes, imaging conditions, and levels of realism or complexity.
- The reviewed dataset section covers X-ray security datasets that are widely used in the literature.
- One dataset contains 15,449 X-ray samples with false-colour material mappings from a dual-energy, four-view scanner.Its samples include multiple object classes and variants for classification and detection.
- GDXray contains 19,407 samples across casting, weld, baggage, natural-image, and settings subsets, with the baggage subset mainly used for security applications.Its non-complex content is described as non-ideal for real-time deployment.
- Another dataset contains 120,000 benign images, with threat images synthetically generated using the TIP algorithm.The images were scanned with a Rapiscan R60 and include separate training and test data.
3.4. SIXray
SIXray provides a large, varied collection for X-ray security tasks, while related datasets support anomaly detection or operational firearm screening but have access constraints.
- SIXray: SIXray contains 1,059,231 X-ray images, including 8,929 manually annotated images spanning six threat classes and background.Objects vary widely in scale and viewpoint and are often overlapping.
- SIXray: SIXray’s varied scale, viewpoint, and overlap make it suitable for real-time classification, detection, and segmentation applications.
- Related datasets: An in-house dual-energy dataset contains 230,275 patches with knife, gun, gun-component, and benign classes for unsupervised anomaly detection.It is not publicly available.
- Related datasets: FFOB contains 4,680 firearm full-weapons and 67,672 operational benign images from commercial screening operations.Its main drawback is restricted access.
3.7. Compass - XP Dataset
The reviewed datasets support X-ray security research across varied object classes, image modalities, and task settings. The listed resources include paired X-ray and photographic data, manually annotated airport-screening images, and a benchmark table for deep-learning datasets.
- Compass-XP Dataset: The Compass-XP dataset contains 501 objects from 369 ImageNet-derived classes and 1,901 paired X-ray and photographic images.Each X-ray image also has low-energy, high-energy, material-density, grayscale, and pseudo-coloured RGB versions.
- Dataset Benchmark: Table 1 benchmarks datasets used in deep-learning applications within X-ray security imaging.The paper states that the best-performing models are explained in subsequent sections.
- OPIXray Dataset: OPIXray contains 8,885 manually annotated airport-screening images covering five sharp-object categories.The dataset includes 7,019 training and 1,776 testing images.
4. Evaluation Criteria
The survey introduces evaluation criteria based on confusion-matrix counts and uses accuracy, rates, mAP, and AUC to assess reviewed methods. Table 1 applies these datasets and criteria to provide benchmark statistics.
- Metric Basis: All evaluation metrics are computed from true positives, false positives, true negatives, and false negatives.These quantities provide the basis for the criteria introduced in this section.
- Accuracy: Accuracy measures correctly predicted samples over all predictions: ACC = (TP + TN)/(TP + TN + FP + FN).It combines correct positive and negative predictions relative to the total number of predictions.
- Rates: True positive rate measures correctly predicted positive samples: TPR = TP/(TP + FN).It captures the proportion of positive samples identified correctly.
- Rates: False positive rate measures negative samples predicted as positive: FPR = FP/(FP + TN).It quantifies false alarms among negative samples.
- Detection Metrics: Mean average precision summarizes average precision, whose area under the precision–recall curve uses precision = TP/(TP + FP) and recall = TP/(FN + TP).mAP is defined as the mean of average precision values.
- Curve Metrics: AUC is the area under the receiver operating characteristic curve plotted from true positive rates and false positive rates.The survey uses it as a classification evaluation criterion.
5. Conventional Image Analysis
Conventional image analysis improves X-ray inputs through enhancement and generates synthetic threat images for training. The reviewed techniques target readability, noise reduction, and threat-image construction.
- Section Scope: Conventional image analysis performs image enhancement and threat-image projection.These are the two broad technique categories introduced in the section.
- Image Enhancement: Low- and high-energy fusion with background subtraction reduces noise, while manual or adaptive thresholding supports image preprocessing.Preprocessing is described as improving image quality and readability for screeners and computers.
- Image Enhancement: Pseudo-colouring assigns colours to grayscale X-ray images to improve detection performance and operator alertness.It is presented as another image-enhancement technique used in the literature.
- Threat Image Projection: Threat image projection creates synthetic training data by multiplying a binary threat mask with a benign X-ray image.Affine or logarithmic transformations can add varied threat projections to benign inputs for human or model training.
6. Machine Learning Approaches in X-ray Security Imaging
Conventional machine-learning research in X-ray security covers classification, detection, and segmentation, with BoVW and related feature-based methods dominating earlier work. Reported performance can be strong, but small datasets constrain scalability for real-time use.
- Section Scope: The survey reviews conventional approaches across classification, detection, and segmentation tasks.It frames this section as a task-based review of machine-learning applications in X-ray security imaging.
- Object Classification: BoVW pipelines extract and cluster visual features with k-means before classification using random forests, SVMs, or sparse representation.This was the prevalent approach before deep learning dominated the field.
- Object Classification: 94.7% vs. 96.3% accuracy was reported for k-NN-based sparse representation and deep models, respectively, on GDXray.The comparison indicates comparable classification accuracy between the conventional and deep approaches in that evaluation.
- Object Detection: Conventional object detection predicts bounding-box coordinates and class labels simultaneously, making the task challenging and relatively limited in the literature.Detection methods primarily employ BoVW, as in classification.
- Object Detection: 46.1% mAP was achieved by the SPIN descriptor with an SVM classifier in conventional X-ray object detection.The result came from evaluating various feature descriptors for BoVW-based detection.
- Object Detection: Multiple-view imagery generally improves detection when rotation and superimposition reduce object visibility from one view.The reviewed multi-stage methods combine feature extraction and classification across views.
- Segmentation: Conventional segmentation progresses from fixed-threshold pixel grouping to nearest-neighbour, background-removal, graph-based, spectral, and variational methods.The literature includes both attribute-relational graphs and newer spectral-clustering or variational approaches.
- Limitation: Small experimental datasets limit the scalability of these conventional techniques for real-time applications.This limitation accompanies the promising detection performance reported in the reviewed studies.
7. Deep Learning in X-ray Security Imaging
Deep learning approaches are organized into supervised classification, detection, and segmentation tasks, with unsupervised anomaly detection also included. Across reviewed studies, CNN-based methods often outperform conventional approaches, while performance depends on input representation, multiple views, scanner domain, and dataset availability.
- The review categorizes deep learning methods into supervised classification, detection, and segmentation, plus unsupervised anomaly detection.
- Figure 4 contrasts image-level classification with ResNet-50, object detection with YOLOv3, and segmentation with Mask RCNN.
- Detection: A proposed CHR model improves ResNet-101 performance on SIXray from 36.01 to 38.14 mAP by addressing class imbalance and clutter.
- Generalisation: Cross-scanner transfer remains challenging because scanner parameters are unknown, with one study reporting DBF3-to-SIXray performance decreasing from 88% to 85% mAP.
- Detection: Detection results include 94.5% average precision, 92.6% recall, and 68 FPS for YOLOv2 on the SASC dataset.
- Detection: Multiple-view imagery improves detection, with an epipolar-constrained YOLOv3 detector outperforming single-view detection by 2.2%.
8. Discussion and Future Directions
The discussion identifies dataset limitations, incomplete use of multiple-view information, scanner-domain transfer challenges, inconsistent material-information results, and limited unsupervised anomaly detection. It points toward larger realistic datasets, improved synthetic-data methods, domain adaptation, and stronger reconstruction techniques.
- Dataset: Large X-ray datasets are scarce or class-biased, limiting deep-model training and reliable supervised methods.
- Dataset: Manual data collection provides realistic and flexible samples but requires substantial human effort, time, and expense.
- Dataset: TIP-generated data can adversely affect detection on real examples, while current GANs cannot produce realistic full X-ray images.
- Multiple-View Information: Multiple-view imagery is valuable when object viewpoints hinder detection, but few studies integrate it within or outside CNNs.
- Domain Adaptation: Transferring models between scanners is challenging because scanner intrinsics are unknown, motivating domain-adaptation research.
- Anomaly Detection: Current unsupervised anomaly-detection performance is limited for real-world deployment, requiring reconstruction techniques that better learn normality.
- Material Information: Material-information studies report inconsistent outcomes, leaving its contribution to accurate classification and detection unresolved.
9. Conclusion
The paper surveys conventional and deep learning methods for X-ray security imaging, organizing deep learning applications across major vision tasks. It identifies dataset, imaging-utilization, and deployment challenges that shape future research.
- The paper taxonomises conventional machine learning and modern deep learning algorithms used in X-ray security imaging.Deep learning coverage includes classification, detection, segmentation, and unsupervised anomaly detection.
- The review covers conventional tasks including image enhancement, threat image projection, segmentation, feature extraction, classification, and detection.
- A lack of large, well-balanced datasets limits development of generalisable deep learning algorithms for real-time deployment.The review also notes that datasets originate from machines with different intrinsic characteristics.
- Most recent approaches do not fully exploit X-ray properties such as multiple-view geometry and high-low energy.Further unsupervised learning could also use existing unlabeled X-ray datasets.
- The paper reviews current techniques’ strengths and weaknesses, discusses open challenges, and outlines future directions for the field.