PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth-Aware Pothole Detection Using YOLO and RT-DETR at the Edge

Md Monjurul Ahsan Prodhan, Md Nour Hossain

arXiv:2608.27633v1cs.CVcs.AIcs.LG

TL;DR

Pothole inspection needs physical depth measurement because RGB-only detection cannot quantify cavity severity, while manual surveys are labor-intensive. The paper compares five RGB-D detection architectures with adverse-condition augmentation and RANSAC orthorectification. YOLOv8n-Seg provides the strongest detection and depth result, while bounding-box models show a consistent structural depth overestimate.

  • Problem

    RGB-only automated approaches cannot measure physical pothole depth, and manual inspection cannot quantify cavity depth for proactive severity assessment.

  • Method

    The framework compares five architectures on PothRGBD using adverse-condition augmentation and RANSAC ground-plane orthorectification before depth statistics are computed.

  • Results

    Bounding-box models overestimate pothole depth by 0.16–0.21 cm relative to segmentation, while YOLOv8n-Seg measures 2.96 cm and YOLOv9t converges to approximately 0.953 mAP@50.

  • Takeaways & Limitations

    Pixel-precise segmentation is the most accurate approach for depth extraction, whereas rectangular bounding boxes introduce a consistent healthy-pavement inclusion bias.

Abstract

from arXiv · show

Pothole detection and its severity measurement is still an important challenges in urban infrastructure management, where late maintenance directly contributes to vehicle damage, road accidents, and escalating repair costs. Existing automated approaches depend on 2D RGB images and cannot measure physical depth of potholes. In this paper, we present a depthaware pothole detection framework and then compare five architectures: YOLOv8n, YOLOv8nSeg, YOLOv9t, RTDETRL, and RTDETRX for RGB-D sensor fusion-based detection and automated depth measurement. A custom offline augmentation pipeline is used here to simulate adverse road monitoring conditions. All models are trained on the PothRGBD dataset with an 80% training and 20% validation split and evaluated using Precision, Recall, mAP@50, and mAP@50_95. Before measuring the depth data, all depth maps are corrected for camera tilt using RANSAC ground-plane orthorectification and all zero-valued sensor pixels are cast to NaN before any statistic is computed. YOLOv8nSeg achieves the highest mAP@50 of 0.9556 and mAP@50_95 of 0.6758 with the most accurate depth estimate of 2.96 cm with the pixel-precise Dseg algorithm. YOLOv8n achieves the fastest inference at 3.6ms. RTDETRX achieves the highest detection confidence at 92.70%. An important finding is that even after full RANSAC orthorectification, bounding box models overestimate pothole depth by 0.16 to 0.21 cm compared to pixel precise segmentation masks. This confirms that the pavement inclusion bias is structural rather than a calibration artifact.

I. INTRODUCTION

Potholes create substantial safety and economic burdens, while conventional inspection cannot quantify cavity depth for proactive severity-based maintenance. The paper addresses this gap with RGB-D depth-aware detection and a comparison of CNN, transformer, and segmentation architectures.

  • 55 million potholes affect nearly four million miles of U.S. roads, with over 30% of urban roads and highways in mediocre condition.
  • £14.4 billion is the estimated annual economic damage from potholes in England.
  • Manual inspection is labor-intensive and cannot provide quantitative cavity-depth measurements for objective repair prioritization.
  • The framework compares YOLOv8n, YOLOv8n-Seg, YOLOv9t, RT-DETR-L, and RT-DETR-X under a unified PothRGBD experimental protocol.
  • The study contributes adverse-condition augmentation, RANSAC ground-plane orthorectification, and quantification of spatial measurement bias.
  • Earlier pothole detectors relied on manual features or RGB imagery, while YOLO enabled end-to-end real-time detection and segmentation methods enabled pixel-precise boundaries.

B. Transformer Architectures for Object Detection

Transformer-based detectors replace or reduce handcrafted detection components through attention and set prediction, while RT-DETR adapts these ideas for real-time inference. RGB-D sensing complements image detection by adding aligned per-pixel depth for infrastructure monitoring.

  • DETR reformulated object detection as direct set prediction using a Transformer encoder-decoder and bipartite matching loss.
  • RT-DETR uses a hybrid encoder with intra-scale interaction and cross-scale fusion to combine efficient feature processing with global attention.
  • RT-DETR eliminates non-maximum suppression, removing a latency and hyperparameter-sensitivity source associated with YOLO-based architectures.
  • RGB-D sensors provide aligned color and depth information as a lower-cost alternative to high-precision LiDAR for infrastructure monitoring.

D. Image Augmentation for Adverse Condition Robustness

Road-monitoring detectors require augmentation that reflects domain-specific visual degradation rather than relying only on generic transformations. The proposed RGB-D pipeline combines adverse-condition simulation with depth-aware processing for physical measurement.

  • Generic cropping, flipping, color jittering, and mosaic augmentation do not fully capture visual degradations in road monitoring.
  • Synthetic weather augmentation combined with real training data has been reported to improve robustness under adverse conditions in autonomous driving.
  • RANSAC fits ground surfaces in 3D point clouds to address systematic camera-pitch effects in raw depth maps.
  • The proposed pipeline performs RGB-D sensor fusion-based pothole detection and physical depth extraction.
  • The PothRGBD dataset contains 1,000 synchronized RGB-depth pairs captured with an Intel RealSense D415 at 640 × 480 resolution.

B. Augmentation Pipeline

The study combines adverse-condition augmentation with five differently sized detection architectures under a common training configuration. The pipeline includes both segmentation and transformer alternatives for RGB-D pothole analysis.

  • Augmentation Pipeline: Six Albumentations techniques simulate precipitation, shadows, glare, low light, and sensor noise for adverse road-monitoring conditions.Each technique is applied at p = 0.4 in the described pipeline.
  • Training Configuration: The five architectures are evaluated under a unified training protocol on augmented training data and a fixed validation split.
  • Augmentation Pipeline: Figure 2 presents an example of the domain-specific augmentation pipeline applied to one PothRGBD pothole image.
  • Model Architectures: YOLOv9t uses PGI and GELAN, while RT-DETR-L employs a hybrid convolutional and global-self-attention encoder.

D. Ground-Plane Orthorectification

The framework corrects RGB-D depth maps for camera tilt before extracting pothole depth, using cleaned point clouds, RANSAC plane fitting, rotation, and reprojection. Detection outputs are then paired with the orthorectified maps for geometry-specific depth estimation.

  • Cleaning and reprojection: Zero-valued depth pixels are cast to NaN before statistics, preventing dead or out-of-range sensor measurements from corrupting depth calculations.The cleaned map is reprojected using approximate Intel RealSense D415 intrinsics.
  • Plane fitting: RANSAC fits the dominant road plane after discarding NaN depths and distances beyond 3,000 mm.The plane uses a 20 mm residual threshold and random state 42.
  • Tilt correction: A Rodrigues rotation aligns the fitted surface normal with the world vertical axis, removing camera pitch and roll.This rotation flattens the road surface perpendicular to the Z-axis.
  • Map reconstruction: The rotated point cloud is interpolated onto the original image grid, producing an orthorectified map whose depth differences represent true vertical distances.The result is independent of how the camera was mounted.
  • Depth extraction: Orthorectified depth maps are fused with detections, using separate extraction algorithms for rectangular boxes and segmentation masks.The distinction reflects their different geometric outputs.
  • Bounding-box extraction: Bounding-box depth uses valid pixels inside the crop and robust percentile statistics rather than raw extrema.Measurements with fewer than 10 valid pixels are reported as sensor errors.

2) Segmentation Mask Depth (YOLOv8n-Seg):

YOLOv8n-Seg estimates pothole depth from a pixel-precise mask and a locally defined surrounding-road baseline. This geometry avoids including healthy pavement inside the pothole region, unlike rectangular bounding boxes.

  • Segmentation mask depth: The segmentation mask is thresholded at 0.5, resized to the original image, and surrounded by a 15-pixel buffered ring for local pavement estimation.The pothole mask is removed from the ring to isolate pavement around the cavity.
  • Geometric rationale: The segmentation formulation is geometric because the mask traces the pothole edge while the surrounding ring contains genuine road-surface pixels.This contrasts with the healthy pavement enclosed by rectangular boxes.
  • Matched comparison: The same RGB-D image pair is used to compare extracted depths from all five models directly.The comparison is conducted after the depth-processing pipeline described elsewhere.

IV. EXPERIMENTS AND RESULTS

The experiments train and evaluate five architectures under a fixed RGB-D protocol on PothRGBD, with detection performance summarized across four metrics. YOLOv8n-Seg leads the reported mAP measures, while YOLOv9t has the highest precision and RT-DETR-L the lowest mAP@50.

  • Experimental setup: 50 epochs at 640 × 640 train each architecture on 1,600 augmented samples and evaluate on the fixed 200-image validation split.All experiments run on a single NVIDIA Tesla T4 with a confidence threshold of 0.25.
  • Evaluation metrics: Precision, Recall, mAP@50, and mAP@50-95 are reported for all five architectures on PothRGBD’s validation set.The evaluation compares the models under a shared metric set.
  • Detection results: 0.9556 mAP@50 and 0.6758 mAP@50-95 are achieved by YOLOv8n-Seg, the highest values across the compared architectures.The result is presented as evidence that its segmentation head improves localization accuracy without sacrificing detection quality.
  • Detection results: 0.9525 Precision is achieved by YOLOv9t, while RT-DETR-L records the lowest mAP@50 at 0.9022.The comparison spans CNN and real-time Transformer architectures.
  • Performance comparison: Figure 5 visualizes all four performance indicators simultaneously across the five assessed models.It presents the models’ multi-dimensional trade-offs at a glance.

C. Training Convergence Analysis

The convergence curves reveal a divide between steadily learning CNNs and transformer models with a prolonged warm-up before rapid improvement. YOLOv8n-Seg converges fastest among the CNNs and reaches the highest reported mAP@50 among them.

  • Figure 6 plots mAP@50 and mAP@50-95 convergence for all five architectures over 50 epochs.
  • YOLOv8n-Seg reaches approximately 0.956 mAP@50 by epoch 50, climbing fastest among the CNN models.The three CNN models begin learning from the first epoch and improve steadily without a prolonged stall.
  • YOLOv9t dips sharply at epoch 2, recovers by epoch 6, and converges smoothly to approximately 0.953.
  • RT-DETR-L and RT-DETR-X remain near zero through the first 12 epochs before jumping sharply around epochs 12–13.Their shaded warm-up region indicates that the models detect almost nothing useful during this initial period.
  • RT-DETR-X surpasses RT-DETR-L from approximately epoch 20 onward, reaching 0.927 by epoch 50 versus RT-DETR-L’s approximately 0.902.

D. Inference Speed Analysis

Inference speed and depth results expose a practical trade-off across architectures: YOLOv8n is fastest, while segmentation provides more accurate physical depth measurement. Bounding-box models consistently overestimate depth even after camera-tilt correction.

  • Inference speed: 3.6 ms and 8.1 GFLOPs make YOLOv8n the fastest architecture measured on the Tesla T4 GPU.YOLOv8n-Seg takes 22.7 ms, while YOLOv9t, RT-DETR-L, and RT-DETR-X take 49.5, 67.3, and 90.2 ms, respectively.
  • RGB-D inference output: RT-DETR-X achieves 92.70% detection confidence on the matched test image used for RGB-D inference comparison.
  • Physical depth measurement: 2.96 cm is the YOLOv8n-Seg depth estimate using Dseg, compared with 3.12–3.17 cm from all four bounding-box models using Dbox.The measurements use the same matched RGB-depth image pair after RANSAC orthorectification and zero-depth masking.
  • Physical depth measurement: 0.16–0.21 cm is the bounding-box depth overestimate relative to pixel-precise segmentation, across lightweight CNN and large Transformer architectures.The bias persists after camera tilt is fully corrected, so it is attributed to the rectangular geometry of bounding boxes.
  • Physical depth measurement: 0.05 cm is the spread among the four bounding-box estimates, ranging from 3.12 cm for YOLOv8n to 3.17 cm for YOLOv9t.Their close agreement supports a consistent systematic bias rather than random measurement noise.

F. Comparative Analysis and Architectural Trade-offs

The five architectures exhibit distinct performance trade-offs rather than a single overall winner. YOLOv8n-Seg leads accuracy and depth estimation, while other models prioritize latency, precision, parameter efficiency, confidence, or recall.

  • YOLOv8n is fastest for real-time edge deployment at 3.6 ms and 8.1 GFLOPs.
  • YOLOv8n-Seg achieves the highest mAP@50 (0.9556), mAP@50-95 (0.6758), and depth accuracy at 2.96 cm via Dseg.
  • YOLOv9t provides the highest Precision (0.9525) with the smallest parameter footprint of 2.00 M.
  • RT-DETR-X achieves the highest confidence at 92.70% and Recall at 0.9208 for pipelines prioritizing detection certainty over latency.
  • No single architecture dominates across all dimensions.

V. CONCLUSION

The study compares five architectures for edge-based, depth-aware pothole detection using PothRGBD and weather augmentation. YOLOv8n-Seg leads detection and depth accuracy, while bounding-box models systematically overestimate depth; future work targets broader validation and deployment.

  • YOLOv8n-Seg achieves the highest mAP@50 of 0.9556 and the most accurate depth estimate of 2.96 cm.
  • YOLOv8n provides the fastest inference at 3.6 ms, while RT-DETR-X achieves the highest confidence at 92.70% and Recall at 0.9208.
  • Bounding-box models overestimate depth by 0.16–0.21 cm relative to segmentation masks, quantifying healthy pavement inclusion bias.
  • Future work will validate the approach on larger datasets, deploy it on embedded hardware such as NVIDIA Jetson, and extend detection to cracking and rutting.
Loading 2608.27633v1…