Source-linked AI summary
Road Damage Detection Using Deep Neural Networks with Images Captured Through a Smartphone
Hiroya Maeda, Yoshihide Sekimoto, Toshikazu Seto, Takehiro Kashiyama, Hiroshi Omata
TL;DR
Road-damage research lacked standardized datasets and often identified damage presence without distinguishing the types needed for practical repair decisions. The study releases a large smartphone-based dataset, trains CNN object detectors, and reports high-accuracy eight-type classification, including smartphone inference results.
Problem
Prior studies lacked an openly available uniform road-damage dataset and often did not distinguish the eight damage types relevant to road managers.
Method
The study creates a large-scale smartphone image dataset and trains and evaluates state-of-the-art CNN-based end-to-end object detection models.
Results
Recalls and precisions greater than 75% were achieved in the best-detectable category with 1.5 s inference on a smartphone, while eight damage types were identified with high accuracy.
Takeaways & Limitations
The dataset, trained models, source code, and smartphone application are publicly available to support road-damage detection research and smartphone-based inspection.
Abstract
from arXiv · showhide
Research on damage detection of road surfaces using image processing techniques has been actively conducted, achieving considerably high detection accuracies. Many studies only focus on the detection of the presence or absence of damage. However, in a real-world scenario, when the road managers from a governing body need to repair such damage, they need to clearly understand the type of damage in order to take effective action. In addition, in many of these previous studies, the researchers acquire their own data using different methods. Hence, there is no uniform road damage dataset available openly, leading to the absence of a benchmark for road damage detection. This study makes three contributions to address these issues. First, to the best of our knowledge, for the first time, a large-scale road damage dataset is prepared. This dataset is composed of 9,053 road damage images captured with a smartphone installed on a car, with 15,435 instances of road surface damage included in these road images. In order to generate this dataset, we cooperated with 7 municipalities in Japan and acquired road images for more than 40 hours. These images were captured in a wide variety of weather and illuminance conditions. In each image, we annotated the bounding box representing the location and type of damage. Next, we used a state-of-the-art object detection method using convolutional neural networks to train the damage detection model with our dataset, and compared the accuracy and runtime speed on both, using a GPU server and a smartphone. Finally, we demonstrate that the type of damage can be classified into eight types with high accuracy by applying the proposed object detection method. The road damage dataset, our experimental results, and the developed smartphone application used in this study are publicly available (https://github.com/sekilab/RoadDamageDetector/).
1 Introduction
The study addresses fragmented road-damage inspection research by releasing a large-scale dataset and evaluating CNN-based object detection for eight damage types.
- A shortage of inspection expertise and resources complicates infrastructure inspection as Japan’s aging structures require increasing scrutiny.
- Existing road-inspection studies lack a common dataset, limiting comparison across results.
- Current methods generally lack end-to-end deep-learning approaches specifically for road-damage detection.
- Many studies detect damage presence or use only a few categories, making their results difficult for road managers to apply directly.
- The study trains and evaluates a state-of-the-art CNN object detector, publishes benchmark results and models, and identifies eight damage types with high accuracy.
- 9,053 images containing 15,435 damages were released with bounding boxes for eight damage types under varied weather and illuminance conditions.
2.1 Road Damage Detection
Road-damage inspection research has progressed from manual and expensive measurement toward automated image-based methods, but practical systems still need broader damage-type detection.
- Visual inspection requires experienced road managers, is time-consuming and expensive, and can be inconsistent and unsustainable.
- Quantitative systems such as mobile measurement and laser scanning provide high accuracy but are considerably expensive.
- Researchers have applied in-vehicle cameras, image processing, and machine learning to automate asphalt-crack and pothole detection.
- Many road-damage methods determine whether damage exists, while only some classify a few damage categories.
- Deep-learning studies have used classification or sliding-window pipelines, whereas end-to-end object detection is reported as more accurate and faster.
- This study applies end-to-end deep-learning object detection to test accuracy and processing speed across eight road-damage classes.
2.2 Image Dataset of Road Surface Damage
Road-damage research lacked a publicly available, standardized dataset, so the study constructs one from smartphone images captured in ordinary vehicles.
- No road-damage-tagged dataset was available, and researchers independently collected images and proposed methods, preventing direct comparison.
- Images taken directly above roads are difficult and costly to reproduce because they require external vehicle cameras and dedicated cars.
- The study publicly releases road-damage images captured by a smartphone on the dashboard of a general passenger car.
- The authors report considerably high detection accuracy even with this simple image-acquisition method.
2.3 Object Detection System
The paper reviews region-based and single-stage object detectors, emphasizing how their architectures trade computation, speed, and detection capability.
- Traditional object detection applies classifiers across varying object sizes and positions, while R-CNN uses region proposals followed by classification.
- Fast R-CNN reduces duplicate computation by processing the entire image once through a feature extractor before classifying crops.
- Faster R-CNN uses a Region Proposal Network and then refines class labels and bounding boxes for each proposal.
- YOLO predicts object regions and classes with a single CNN and is designed for high mean average precision and speed.
- R-FCN uses a region-based fully convolutional architecture to support accurate and efficient detection while reducing region-specific computation.
- SSD directly predicts classes and anchor offsets with one feed-forward convolutional network using multi-scale feature-map outputs.
2.4 Base Network
The study reviews representative convolutional base networks for object detection, emphasizing differences in architecture, depth, computational cost, and suitability for mobile inference.
- Seven representative base networks were selected, including widely used computer-vision feature extractors.The supplied passage names six extractors and indicates that three were evaluated in Section 5.
- Darknet-19 is the YOLO base model and contains 19 convolutional layers and 5 maxpooling layers.
- VGG 16 contains 13 convolutional and 3 fully connected layers within a 16-layer CNN.
- ResNet enables high-precision learning in very deep CNNs and achieved a 3.57% ImageNet test-set error rate with 154 layers.
- Inception units increase network depth and breadth without increasing parameter count or computational complexity, while Inception-ResNet combines them with residual connections.
- MobileNet targets efficient mobile inference by using depthwise separable convolutions that reduce computational cost and parameter count.It reportedly achieves accuracy comparable to VGG-16 on ImageNet with 1/30th of the computational cost and model size.
3 Proposed Dataset
The proposed dataset uses smartphone images captured from a vehicle across diverse Japanese municipalities and annotates road damage with bounding boxes and class labels. It covers eight damage categories and is designed for practical, publicly accessible road inspection research.
- The dataset addresses the practical advantage of vehicle-mounted imaging over overhead capture for road-damage model development.
- Images were collected from 163,664 road images across seven Japanese local governments with diverse regional, weather, and fiscal conditions.The collection covered approximately 1,500 km in total.
- A dashboard-mounted smartphone captured 600 × 600-pixel images once per second while recording location information.The one-second interval was selected for approximately 40 km/h travel, or about 10 m/s, to avoid leakage or duplication.
- The dataset defines eight damage categories, including linear cracks, alligator cracks, potholes, rutting, and white-line blurring.
- The study reports broader damage coverage than prior image-processing research, which often detected only potholes or limited crack orientations.
- Manual annotation follows a PASCAL VOC-like format, drawing each damage bounding box before attaching its class label.The format is intended to support application with existing image-processing methods.
- The released dataset contains 9,053 labeled images and 15,435 damage bounding boxes at 600 × 600 resolution.Its regions and weather conditions are diverse, although the authors note unavoidable data bias, including few D40 instances.
- The public dataset blurs clearly visible faces and license plates to address privacy concerns.
4 Experimental Setup
The experiments train SSD Inception V2 and SSD MobileNet detectors on the proposed dataset, using an 8:2 training–evaluation split and smartphone-oriented model settings.
- SSD Inception V2 and SSD MobileNet were selected because prior comparisons reported relatively small CPU loads and low memory consumption while maintaining high accuracy.The study evaluates these frameworks on its own road-damage dataset rather than relying only on COCO results.
- The dataset materials include eight-category sample images and a table defining the road-damage classes.
- The annotation pipeline first draws a bounding box and then attaches the corresponding class label.
- The SSD models use 300 × 300 inputs, resizing the original 600 × 600 images before detection.
- The dataset was randomly split 8:2 into 7,240 training images and 1,813 evaluation images.
5 Results
The experiments compare SSD Inception V2 and SSD MobileNet across damage detection accuracy and inference speed on a PC and Nexus 5X smartphone. SSD MobileNet performs better overall and supports smartphone processing in 1.5 seconds.
- D01 and D44 achieve relatively high recall and precision, whereas D11 and D40 have low recall associated with their training-data counts.
- D43 achieves high recall and precision despite limited training data because its striped pedestrian-crossing blur occupies a large, clear image region.
- Damage-instance distributions differ across municipalities; Muroran has 1,192 D20 damages, attributed to alligator cracks during snow thaw.
- SSD MobileNet yields better overall results than SSD Inception V2.
- SSD Inception V2 is two times slower than SSD MobileNet in the inference-speed comparison.
- The smartphone processes data in 1.5 s, enabling real-time road-surface damage detection in a moving car with the accuracy reported in Table 2.
6 Conclusions
The study developed and released a large-scale, eight-class road-damage dataset and evaluated a deep-learning damage detection model. The best-detectable category achieved recalls and precisions greater than 75% with 1.5 s smartphone inference.
- 163,664 road images were collected in collaboration with seven local governments in Japan.
- 9,053 images were annotated and released as a training dataset after road damage was visually confirmed and classified into eight classes.
- The dataset was presented as the first road-damage detection dataset and was made publicly available with trained models, source code, and a smartphone application.
- Recalls and precisions greater than 75% were achieved in the best-detectable category with an inference time of 1.5 s on a smartphone.