Source-linked AI summary
AUTOPILOT An Advanced Perception, Localization and Path Planning Techniques for Autonomous Vehicles Using YOLOv7 and MiDaS
Harshkumar Devmurari, Gautham Kuckian, Prajjwal Vishwakarma
TL;DR
Autonomous driving requires advances in perception, localization, decision-making, and control, and this project studies a combined approach to those challenges. It uses YOLO, MiDaS, perspective transformation, and path planning; the reported results include object-detection precision and recall and a depth-estimation error. The paper also notes monocular-depth scale ambiguity.
Problem
Self-driving cars require advances in perception, localization, decision-making, and control, while robust perception and localization in changing environments remain challenges.
Source: §I · p. 1Method
The project combines YOLO object detection, MiDaS depth sensing, perspective transformation for localization, and A* path planning.
Source: §I · p. 1, C. Localization · p. 3Results
Object detection achieved mean precision of 89% and mean recall of 92%, while depth estimation had an average zero shot error of 8.6.
Source: A. Object Detection · p. 4, B. Depth Sensing · p. 4, §VI · p. 6Takeaways & Limitations
The combined approach is presented as an alternative solution for object detection, depth sensing, localization, and path planning in self-driving systems.
Source: §I · p. 1, §VI · p. 6Monocular MiDaS depth estimation has scale ambiguity without an absolute scale reference, making true object size or absolute distance difficult to determine.
Source: A. Limitations And Challenges Of the… · p. 5
Abstract
from arXiv · showhide
Self driving vehicles have emerged as a reliable technology that has the capability to transform transportation and mobility. The development of self driving cars requires significant advances in a number of areas, including perception, localization, decision making, and control. This research paper is based on the project implementation of the combination of object detection using YOLO (You Only Look Once), depth sensing using MiDaS for the localization and perception of obstacles, perspective transform, and decision making for path planning in self driving cars. The contemporary state of the technology for object detection, depth sensing, localization, and path planning evaluates the performance of the combined system through simulations and experiments. The results show that the combination of YOLO and MiDaS provides a new robust system for object detection and depth sensing. This research paper contributes to the advancement of self driving car technology and provides new and innovative approaches to the perception and localization of obstacles in the environment. Keywords: YOLO, MiDaS, perception, localization, decision making
I. INTRODUCTION
The introduction frames autonomous driving as requiring coordinated perception, localization, path planning, and control, and describes this project’s combination of YOLO, MiDaS, perspective transformation, and path planning. It situates the approach among prior work on real-time detection, depth estimation, camera calibration, and 3D recognition.
I. INTRODUCTION
- The project combines YOLO object detection, MiDaS depth sensing, perspective transformation, and path planning, and evaluates the system through simulations and experiments.Its stated scope covers perception, localization, and path planning in a full self-driving-car architecture.Source: §I · p. 1
A. Overview Of Existing Research
- YOLOv4 uses larger inputs, enhanced data augmentation, false-positive reduction, and CSPDarknet, and is reported to improve benchmark accuracy while retaining real-time performance.The cited overview also attributes faster training and inference to CSPDarknet.Source: A. Overview Of Existing Research · pp. 1–2
- YOLO detects objects from full images with one convolutional neural network and achieved 63.4% average precision on VOC 2012, outperforming other methods in speed and accuracy.The passage reports real-time detection at 45 frames per second.Source: A. Overview Of Existing Research · p. 2
- A single-RGB-image depth method was proposed for handheld devices because stereo cameras and structured-light sensors were considered unsuitable for that setting.The cited work also concerns depth-aware AR, real-time use, and smartphone image blurring.Source: A. Overview Of Existing Research · p. 2
- A camera-calibration method estimates focal length and principal point from a known quadrilateral’s single-view projection, using perspective projection and planarity.Source: A. Overview Of Existing Research · p. 2
- A YOLO-based 3D recognition approach using RGB and depth surpassed state-of-the-art methods on NYUv2-3D, while a review discusses detection components and challenges including occlusion and scale variation [11].The 3D approach reports mean precision of 82.8% and mean recall of 73.5% on NYUv2-3D.Source: A. Overview Of Existing Research · p. 2
B. Characteristics Of Advance Self Driving Cars
This section presents YOLOv7 and MiDaS as technologies for object detection and depth estimation in self-driving cars, alongside perspective transformation and path-planning algorithms. It also notes unresolved technical and societal challenges.
B. Characteristics Of Advance Self Driving Cars
- YOLOv7 is described as enabling real-time object detection and tracking for responses to changing traffic conditions.Source: B. Characteristics Of Advance Self… · p. 2
- MiDaS is described as estimating depth from a single monocular image and producing real-time depth maps for environmental perception.Source: B. Characteristics Of Advance Self… · p. 2
- Combining detection, depth estimation, perspective transformation, and path planning is presented as having the potential to enhance road safety and reliability [13][14].Source: B. Characteristics Of Advance Self… · p. 2
- Challenges before full self-driving deployment include improving model accuracy and robustness, overall system speed, data privacy, and testing and validation frameworks.Ethical and societal considerations include collision accountability, employment, urban planning, and laws and norms for responsible deployment.Source: B. Characteristics Of Advance Self… · p. 2
C. Analysis And Limitations Of the Existing System
The discussion identifies data, environmental, computational, integration, and regulatory hurdles for deep-learning-based autonomous driving. It emphasizes that object detection, localization, and path planning must be integrated with vehicle control.
C. Analysis And Limitations Of the Existing System
- Deep-learning object detection requires substantial labeled data, can struggle in challenging lighting and weather, and may demand computational resources that limit use on low-power embedded systems [16].Source: C. Analysis And Limitations Of the… · p. 2
- Models require continuing updates for changing environments and new objects, while integrating detection, localization, path planning, and precise vehicle control remains complex.Source: C. Analysis And Limitations Of the… · p. 3
- The discussion identifies technological, practical, and regulatory hurdles that remain before fully autonomous driving can be achieved.Source: C. Analysis And Limitations Of the… · p. 3
III. METHODOLOGY
The methodology divides the project into sequential stages covering perception, occupancy-grid construction, and optimal-path planning. It uses separate datasets for object detection and depth estimation.
III. METHODOLOGY
- The project proceeds through perception, occupancy-grid plotting, and optimal-path planning, alongside data collection, model training, localization, and path planning.Source: §III · p. 3
A. Data Collection
- The project uses separate datasets for its two machine-learning models: COCO for object detection and MiDaS trained on 12 datasets for depth estimation.COCO contains 80 classes of common objects and is divided into training and test splits.Source: A. Data Collection · p. 3
B. Training Machine Learning Models
The model uses pretrained YOLOv7 weights and MiDaS weights, while localization transforms image-space detections into a top-view occupancy map. The described setup balances detection training with real-world spatial representation.
B. Training Machine Learning Models
- YOLOv7 is trained on COCO for 100 epochs with batch size 16, using 30% IoU and 25% confidence thresholds.The model is trained with transfer learning from pretrained weights; the best weights are saved after each epoch.Source: B. Training Machine Learning Models · p. 3
C. Localization
- Image-space detections must be transformed into object space with real-world distance for vehicle localization.Source: C. Localization · p. 3
- Perspective transform converts the front view into a bird’s-eye view and a 500x500 occupancy map for estimating object distances.Front-view coordinates are passed through perspective-transform matrices to create the map.Source: C. Localization · p. 3
D. Path Planning
The section reports object-detection performance, describes A* path planning, and identifies varied driving conditions as the evaluation setting.
D. Path Planning
- A* with a minimum-distance heuristic was selected after performing comparatively better than the other algorithms and is used to traverse to a top-view goal point.Source: D. Path Planning · p. 3
IV. RESULTS
- The evaluation dataset was collected in tests spanning different driving conditions, including varied lighting and weather.Source: §IV · p. 3
A. Object Detection
- 93% precision and 95% recall in clear conditions fell to 83% precision and 87% recall in low-light, rainy conditions; mean precision was 89% and mean recall was 92%.The system ran at 80 frames per second on a GeForce RTX3060.Source: A. Object Detection · p. 4
B. Depth Sensing
The depth-sensing system is reported to run in real time, while localization uses a 500x500 grid selected to balance path quality and computational demands.
B. Depth Sensing
- The figures are identified as MiDaS depth perception and top-view localization, respectively.Source: Figure 3 · p. 4, Figure 4 · p. 4
- Depth sensing ran at an average 74 FPS and had an average zero-shot error of 8.6 across varying lighting and weather conditions.Source: B. Depth Sensing · p. 4
C. Localization
- A 500x500 occupancy grid was found best suited to the system’s accuracy-latency trade-off; smaller grids reduced path quality, while larger grids increased computation beyond real-time suitability.Source: C. Localization · p. 4
D. Path Planning
The path-planning section compares four algorithms using path and execution measures across simulated surroundings with varying obstacle densities.
D. Path Planning
- A*, BFS, Dijkstra, and DFS are compared across test cases representing surroundings with varying obstacle densities.Source: D. Path Planning · p. 5
- The comparison measures path length, execution time, expanded-node count, and maximum turning angle.Source: D. Path Planning · p. 5
- The results are presented under a table titled “Algorithm Performance Comparison.”Source: Table I · p. 5
V. DISCUSSION
The discussion frames safety and reliability as ongoing concerns and identifies constraints across perception, planning, testing, and ethical decision-making.
A. Limitations And Challenges Of the Research
- The COCO-trained detector covers 80 classes; objects outside the dataset may go undetected and result in collision.The paper states that the model can detect only the 80 object classes on which it was trained.Source: A. Limitations And Challenges Of the… · p. 5
- Monocular MiDaS depth estimation lacks an absolute scale reference, making true object size or absolute distance difficult to determine.The passage also notes that monocular depth estimation relies on image features and is prone to scale ambiguity.Source: A. Limitations And Challenges Of the… · p. 5
- A single front-facing camera has limited field of vision and cannot cover the full 360° around the car.The paper says this restricts detection of objects and road conditions outside the camera’s range.Source: A. Limitations And Challenges Of the… · p. 5
- Without a global environmental map, the system is limited to local path planning and assumes a hardcoded goal at the horizon.The paper identifies integration with global path plans as necessary for real-world applications.Source: A. Limitations And Challenges Of the… · p. 5
- Testing in controlled conditions may not reflect the complexity and unpredictability of real-world driving scenarios.The paper notes that task-specific simulations do not reproduce all unpredictable conditions vehicles may face after deployment.Source: A. Limitations And Challenges Of the… · p. 5
- System failures could have serious consequences, and public-road testing presents significant risk and requires extensive safety protocols and regulations.The paper describes safe navigation as an ongoing challenge.Source: A. Limitations And Challenges Of the… · p. 5
- The research raises ethical questions about allocating responsibility after accidents and prioritizing occupants versus pedestrians when harm is at stake.The passage also identifies moral dilemmas in autonomous-car decision-making.Source: A. Limitations And Challenges Of the… · p. 5
B. Suggestions For Future Research
The future-research suggestions span localization, system speed, adverse-weather handling, and integration with existing transportation infrastructure.
B. Suggestions For Future Research
- Integrating SLAM or visual odometry and fusing multiple sensor sources are proposed to improve localization robustness.The suggested aim is more accurate and resilient location estimates in complex environments.Source: B. Suggestions For Future Research · p. 5
- Future work should develop a faster, lower-latency overall architecture to handle sensor data, models, algorithms, and decisions within driving’s time constraints.The passage identifies processing time as crucial for autonomous driving.Source: B. Suggestions For Future Research · p. 5
- Future research should improve performance reliability in adverse weather, including fog, snow, and rain.The passages call for algorithms and sensor systems that support navigation across these weather conditions.Source: B. Suggestions For Future Research · pp. 5–6
- Research should develop strategies for autonomous vehicles to coexist with traditional vehicles, pedestrians, and cyclists and interact with traffic systems.Suggested infrastructure work includes communication protocols for traffic signals, smart intersections, and other traffic-management systems.Source: B. Suggestions For Future Research · p. 6
VI. CONCLUSION
The paper combines YOLO, MiDaS, perspective transformation, and A* for autonomous-driving perception and path planning. It reports object-detection scores and depth error, while noting susceptibility to weather and camera-quality factors.
VI. CONCLUSION
- 89% average precision, 92% average recall, and an average zero shot error of 8.6 are reported; the depth error is slightly better than rates from previous MiDaS versions.The authors describe the object-detection results as promising and say detections were accurate in most cases.Source: §VI · p. 6
- The approach fuses YOLO, MiDaS, perspective-transformation localization, and A* routing for path planning.The authors present this combination as an alternative approach to common self-driving-system problems.Source: §VI · p. 6
- The model remains susceptible to adverse weather and camera-quality limitations; proposed future work includes optimization, feature-extraction methods, and newer technologies.The authors describe the approach as a benchmark for future autonomous-vehicle development.Source: §VI · p. 6