Computer Vision for Road Imaging and Pothole Detection: A State-of-the-Art Review of Systems and Algorithms

Introduction

• Road potholes threaten vehicle conditions and traffic safety, making frequent inspection and repair necessary.The passage cites 11,706 complaints in Chicago during the first two months of 2018.Source: Introduction · p. 1
• Manual visual inspection remains the main detection approach, but it is inefficient, expensive, dangerous, and subjective.Automated systems are sought to reconstruct, recognize, and localize potholes efficiently, accurately, and objectively.Source: Introduction · p. 1
• The article reviews road imaging systems, pothole-detection algorithms, open-access datasets, and challenges to address a gap in surveys of cutting-edge techniques.It categorizes detection approaches as classical 2-D image processing, 3-D point cloud modeling and segmentation, machine/deep learning, and hybrid methods.Source: Introduction · p. 2
• The review aims to guide development of next-generation 3-D road imaging and pothole-detection algorithms.Source: Introduction · p. 2

Road Imaging Systems

• Road imaging is typically the first step of intelligent inspection; unrelated 2-D images cannot show road-surface geometry and image segmentation can suffer in poor illumination.The commonly used 3-D acquisition sensors include laser scanners, Microsoft Kinect sensors, and stereo cameras.Source: Road Imaging Systems · p. 2, Figure 2 · p. 3
• Laser scanners use trigonometric triangulation for accurate 3-D road measurements but require dedicated inspection vehicles and costly purchase and maintenance.The receiver is positioned a known distance from the laser source, and reflection angle is used to calculate measurements.Source: Road Imaging Systems · pp. 2–3
• Microsoft Kinect sensors are convenient and cost-effective, but direct sunlight can saturate infrared sensing and road-surface reconstruction accuracy is unsatisfactory.Source: Road Imaging Systems · p. 3
• Multiple road images from different views, captured by a movable camera or synchronized camera array, support 3-D reconstruction through multi-view geometry.Stereo matching estimates depth from horizontal disparities between visual feature correspondences in synchronously captured images.Source: Road Imaging Systems · p. 3
• Reconstructed 3-D road geometry from stereo cameras is reported with accuracy over 3 mm, and stereo cameras are cheaper and more reliable than laser scanners and Microsoft Kinect.CNNs have demonstrated greater disparity-estimation results than traditional explicit-programming methods.Source: Road Imaging Systems · p. 3

Road Pothole Detection Approaches

• The review groups approaches into classical 2-D processing, 3-D point-cloud modeling and segmentation, machine/deep learning, and hybrids combining categories.Machine/deep learning addresses image classification, object recognition, or semantic segmentation; 3-D methods compare observed point clouds with fitted geometric models.Source: Road Pothole Detection Approaches · p. 4

Classical 2-D Image Processing

• Classical 2-D detection typically uses four stages: image preprocessing, segmentation, damaged-area extraction, and result post-processing.The last two stages are typically performed jointly.Source: Classical 2-D Image Processing · pp. 4–5
• Preprocessing filters reduce redundant information and highlight damaged areas; disparity transformation can make damage more distinguishable in road images.Examples include median, Gaussian, bilateral, and morphological filtering; [6] provides a closed-form solution to the disparity transformation optimization problem described in [3] and [50].Source: Classical 2-D Image Processing · pp. 4–5
• Thresholding methods segment damaged foreground from undamaged background, and recent approaches report more effective segmentation on transformed disparity images.The review describes Otsu’s, triangle, and adaptive thresholding, as well as SLIC superpixels followed by an adaptive threshold.Source: Classical 2-D Image Processing · p. 5
• Extraction and post-processing use assumptions about pothole shape, texture, and intensity, with examples including ellipse-based texture checks and geometric-feature classification.Potholes are typically treated as concave, coarser or grainier than surrounding road, and darker due to shadows.Source: Classical 2-D Image Processing · p. 5
• Classical 2-D methods rely on early techniques and can be severely affected by environmental factors; the review says modern 3-D and machine-learning methods have overcome these shortcomings.Source: Classical 2-D Image Processing · p. 5

3-D Point Cloud Modeling and Segmentation

• Point-cloud methods fit a planar or quadratic surface to observed 3-D road points, then segment points by comparison with that model.Examples use least-squares fitting and connected-component labeling, or planar fitting followed by K-means clustering and region growing.Source: 3-D Point Cloud Modeling and… · pp. 5–6
• Outliers can impair least-squares surface fitting, prompting robust alternatives; laser and LiDAR methods also use road profiles or distances to a fitted plane.The review describes bi-square weighted fitting, RANSAC, and surface-normal information; it notes that lacks algorithm details and quantitative road-damage results.Source: 3-D Point Cloud Modeling and… · p. 6

Machine/Deep Learning

• Deep CNNs learn from human-annotated road data and are organized into image-classification, object-detection, and semantic-segmentation networks.Classification identifies positive or negative road images, object detection localizes pothole instances, and semantic segmentation produces pixel- or semantic-level results.Source: Machine/Deep Learning · p. 7

Image Classification-Based Methods

• DCNNs learn hierarchical features and improve on traditional SVM approaches; the review considers image-classification pothole detection a well-solved problem.The reviewed studies include networks trained from scratch and established backbones, with reported strengths including classification under different illumination and thermal images in night or fog.Source: Image Classification-Based Methods · p. 7

Object Detection-Based Methods

• Object detectors include SSD, R-CNN, and YOLO families; the review reports Faster R-CNN with ResNet-101 as the best overall among four tested networks in [75].YOLO methods predict bounding boxes over image grids, while object-detection approaches provide instance-level rather than pixel-level results.Source: Object Detection-Based Methods · pp. 7–9

Semantic Segmentation-Based Methods

• Semantic segmentation includes single-modal and data-fusion networks; approaches also use pseudo-labels, attention modules, few-shot learning, and GAN-generated training images.The review notes that SoTA semantic-segmentation networks require considerable data and identifies unsupervised or self-supervised learning as an area needing more attention.Source: Semantic Segmentation-Based Methods · pp. 9–10
• AA-UNet and AA-RTFNet outperform other SoTA single-modal and data-fusion semantic-segmentation networks, respectively; GAN-based augmentation improves accuracy and accelerates convergence.GAL-DeepLabv3+ outperforms other SoTA single-modal semantic-segmentation DCNNs.Source: Semantic Segmentation-Based Methods · p. 10

Hybrid Methods

• Hybrid methods combine categories, including 2-D keyframe selection with 3-D modeling, image processing with learned classifiers, and semantic image segmentation with point-cloud analysis.The review says hybrids have been studied for over a decade and brought SoTA results; uses exterior pothole edges to fit planes and estimate volumes, while interior edges help reduce incorrect detections.Source: Hybrid Methods · pp. 10–11

Public Datasets

• Available datasets include 367 healthy and 357 pothole training images for classification, and a separate instance-detection set with 2,658 healthy and 1,119 pothole training images.The classification dataset has eight test images per category; the instance-detection dataset has 628 test images captured with a GoPro Hero 3+.Source: Public Datasets · p. 11
• An Indian-road semantic-segmentation dataset contains 2,475 training and 752 test images annotated for road, pothole, footpath, shallow path, and background.The images have 720 × 1280 pixel resolution.Source: Public Datasets · pp. 11–12
• The collection includes varied road-damage datasets, including 9,053 Japanese road images with 15,435 damages and 665 image/ground-truth pairs from different road conditions.A separate binary-classification dataset contains 352 undamaged-road images and 329 pothole images and is described as small.Source: Public Datasets · p. 12
• Multimodal resources include 55 groups of color, subpixel disparity, transformed disparity, and pixel-level annotations, plus Pothole-600 color and transformed-disparity images.CCSAD provides 500 GB of high-resolution stereo images with IMU and GPS data across four driving scenarios.Source: Public Datasets · p. 12

• Classical 2-D methods are computationally intensive, sensitive to illumination and weather, and rely on geometric assumptions that irregular potholes can violate.Source: Existing Challenges and Future Trends · p. 12
• 3-D point-cloud methods assume a single-frame road surface is planar or quadratic, restricting field of view; extensive parameter requirements challenge adaptation to new scenarios, [3].Source: Existing Challenges and Future Trends · p. 12
• Image classification detects pothole presence and object detection gives instance-level results, but neither provides the geometric properties that motivate hybrid 3-D reconstruction and semantic segmentation.Source: Existing Challenges and Future Trends · p. 12
• Large labeled datasets are difficult to prepare for supervised stereo matching and semantic segmentation, motivating road-specific un/self-supervised stereo and few/low-shot segmentation.Source: Existing Challenges and Future Trends · p. 12
• Data-fusion segmentation is computationally complicated; pruning, distillation, and image-classifier keyframe selection are proposed ways to address computation, while pothole rarity also motivates few/low-shot segmentation.Source: Existing Challenges and Future Trends · p. 12

• The review covers state-of-the-art road imaging techniques and computer-vision algorithms for pothole detection, with emphasis on well-performing DCNNs because earlier 2-D and 3-D approaches have serious limitations.Source: Conclusion · p. 12
• Hybrid approaches combining stereo-matching-based 3-D road reconstruction and data-fusion semantic segmentation are identified as a future direction because transportation departments prioritize pothole geometry.Source: Conclusion · p. 12
• Large human-annotated datasets are labor-intensive to prepare, so the review highlights road-specific un/self-supervised stereo and few/low-shot semantic segmentation as research priorities.Source: Conclusion · p. 12