PaperlayerView exact revision on arXiv

Source-linked AI summary

Caroline: An Autonomously Driving Vehicle for Urban Environments

Fred W. Rauskolb, Kai Berger, Christian Lipski, Marcus Magnor, Karsten Cornelsen, Jan Effertz, Thomas Form, Fabian Graefe, Sebastian Ohl, Walter Schumacher, Jörn Marten Wille, Peter Hecker, Tobias Nothdurft, Michael Doering, Kai Homeier, Johannes Morgenroth, Lars Wolf, Christian Basarke, Christian Berger, Tim Gülke, Felix Klose, Bernhard Rumpe

arXiv:1409.6584v1cs.RO

TL;DR

Urban autonomy calls for vehicles that can interpret surroundings and support safe driving, while urban lane detection remains difficult. This paper presents Caroline’s sensor-fusion and decision architecture; in the final event, it drove approximately 17 kilometers in about three hours.

  • Problem

    Urban driver assistance requires understanding surroundings and evaluating risks, while detecting lane markings on urban roads is difficult.

    Source: §1 · p. 2, §4.2 · p. 22

  • Method

    Caroline combines sensor fusion, a digital map, and DAMN-based artificial intelligence to generate driving decisions from sensed surroundings and mission information.

    Source: §3 · p. 9

  • Results

    Caroline drove approximately 17 kilometers in about three hours during the DARPA Urban Challenge Final Event.

    Source: §7 · p. 67

  • Takeaways & Limitations

    The reported run demonstrates Caroline’s ability to perform missions autonomously in urban environments.

    Source: §7 · p. 67

    Grid updates assume obstacles block optical rays, so cells along a sensor ray are treated as lower than the ray itself.

    Source: §4.1 · p. 20

Abstract

from arXiv · show

The 2007 DARPA Urban Challenge afforded the golden opportunity for the Technische Universit\"at Braunschweig to demonstrate its abilities to develop an autonomously driving vehicle to compete with the world's best competitors. After several stages of qualification, our team CarOLO qualified early for the DARPA Urban Challenge Final Event and was among only eleven teams from initially 89 competitors to compete in the final. We had the ability to work together in a large group of experts, each contributing his expertise in his discipline, and significant organisational, financial and technical support by local sponsors who helped us to become the best non-US team. In this report, we describe the 2007 DARPA Urban Challenge, our contribution "Caroline", the technology and algorithms along with her performance in the DARPA Urban Challenge Final Event on November 3, 2007.

1 Motivation and Introduction

The 2007 DARPA Urban Challenge was intended to advance unmanned vehicles for urban settings, with potential benefits for driver assistance and vehicle safety. CarOLO’s work also focused on the engineering process, testing, and a longer-term research basis for reliable assistance systems.

Motivation and Introduction

  • The Urban Challenge aimed to advance unmanned vehicles for urban areas, with potential technology and engineering spin-offs for autonomous vehicles and driver assistance.The introduction describes driver-assistance functions as understanding surroundings, evaluating risks, and helping drivers behave safely and, if desired, efficiently.Source: §1 · p. 2
  • Developing Caroline required coordinated methods, tools, and processes across disciplines, including simulation to cover traffic situations that would be difficult to test in reality.The authors identify the deadline-oriented development of a reliable, complex system as a central project challenge.Source: §1 · p. 2
  • CarOLO’s longer-term goal was to establish a basis for research on vehicle safety and reliable driver assistance, especially sensor fusion, system architectures, simulation, and testing.The interdisciplinary team drew members from five institutes and received support from a company consortium.Source: §1 · pp. 2–3
  • The paper proceeds from challenge requirements and system architecture through technical modules and development processes to Caroline’s qualification and final-event performance.The authors also identify quality assurance and simulator-supported test coverage as topics of the development-process section.Source: §1 · p. 3

2 2007 DARPA Urban Challenge

The 2007 Urban Challenge extended DARPA’s autonomous-vehicle challenges from desert driving to urban missions. CarOLO entered as a Track B team, passed successive qualification stages, and qualified early for the final event.

2007 DARPA Urban Challenge

  • The 2007 challenge shifted the task toward urban environments after the 2004 and 2005 events focused on desert courses.The 2004 course ran about 142 miles from Barstow to Primm, but no vehicle came close to completing it; in 2005, five teams completed the 131.6-mile course.Source: §2 · pp. 3–4
  • CarOLO entered as a Track B competitor with its 2006 Volkswagen Passat, supported by five university institutes and industrial partners.Track B teams received no DARPA development funding and submitted a five-minute vehicle-capability video for selection to the Site Visit stage.Source: §2 · p. 4
  • Caroline demonstrated lane keeping, intersection precedence, passing, and safe behavior at the Site Visit despite a thunderstorm, then received an invitation to the National Qualification Event.The Site Visit demonstration took place at Southwest Research Institute in San Antonio on June 20.Source: §2 · p. 5
  • Caroline qualified early for the November 3 final after demonstrating performance across Track A, B, and C qualification courses.Track A tested merging into moving traffic, Track B tested long routes with stationary obstacles, and Track C tested intersections and blocked roads.Source: §2 · pp. 5–6

3 System Architecture

Caroline used a modular, mostly linear software architecture to separate sensor acquisition, perception, mapping, decision-making, and control. Its sensor streams supported both obstacle detection and estimation of road drivability.

System Architecture

  • Front and rear sensors detect obstacles up to 200 meters away, while four cameras detect and track lane markings for lane keeping.Stereo vision, a color camera, and roof-mounted laser scanners provide information about terrain drivability; rear sensors include radar-based blind-spot detectors.Source: §3 · p. 6
  • Eight principal modules with predefined interfaces and mostly linear signal flow were designed to support independent development with minimal interference.The modules include sensor acquisition and fusion, image processing, a digital map, artificial intelligence, path planning and low-level control, watchdog and diagnosis, and telemetry and offline data storage.Source: §3 · p. 7, Figure 5 · p. 8
  • Image data is processed on acquisition PCs; camera and laser measurements are fused into a drivability grid and passed to artificial intelligence.Lane-detection data is sent directly to the artificial intelligence module.Source: §3 · p. 9
  • Sensor fusion sends object-based surroundings to a digital map that combines online observations with mission and route files for DAMN-based driving decisions.The system uses a Distributed Architecture for Mobile Navigation scheme, as proposed by.Source: §3 · p. 9
  • A soft real-time controller generates and optimizes trajectories before hard real-time control sends commands to the vehicle actuators.The architecture runs on automotive PCs linked over Ethernet, with access to by-wire steering, braking, and throttle through a CANLOG III interface that also connects to the E-stop system.Source: §3 · pp. 6, 7, 9
  • A central watchdog can kill and restart processes, computers, or sensors independently, while visualization, recording, and playback support development and offline analysis.The watchdog supervises the system modules; the visualization module displays exchanged objects such as obstacles, lanes, terrain drivability, and planned paths.Source: §3 · p. 9

4 System Modules

This section describes Caroline’s system modules for environment perception and assessment, motion planning and vehicle control, and safety supervision. It covers sensor fusion and vision, planning and control methods, and the lateral controller’s test performance and safety systems.

4.1 Sensor Fusion

  • Sensor fusion separates object-oriented obstacle tracking from raw-measurement grid fusion, distributing front and rear object processing and grid processing across three computing units.The grid unit processes raw readings from the SICK scanners, stereo vision system, and mono camera.Source: §4.1 · pp. 10–12
  • Track management removes stale tracks, merges or splits tracks when objects converge or separate, and uses graph-based partitioning to split tracks containing independent objects.Clients receive track create, update, and delete messages rather than full database snapshots, reducing network load without information loss.Source: §4.1 · pp. 13, 17
  • A two-stage association process first screens track–measurement matches, then optimizes contour-point matches, while pretracking uses sensor redundancy to set track-activation thresholds.This addresses the risk that incorrect initialization and data association produce faulty tracks.Source: §4.1 · pp. 13–15
  • The multi-point, modelswitching EKF tracks spread contours without a stable object reference point at no significantly higher computational effort than a standard EKF.It uses contour-point matches to update shared motion values while avoiding separate Kalman gains for each point.Source: §4.1 · pp. 15, 17
  • A rolling grid keeps memory and processing manageable by shifting its reference with the vehicle and clearing subgrids at the new horizon.A 100 × 100 m view at 25 cm resolution would require 160,000 cells and about 256 MBit/s at 10 Hz under the described assumptions.Source: §4.1 · pp. 18–19
  • Grid fusion transforms laser and stereo point clouds into the earth-fixed frame, updates cells along traced rays, and combines drivability classifications using Dempster-Shafer masses.Stereo classifications map to mass sets, while mono vision maps pixel drivability values to drivable, undrivable, and unknown masses.Source: §4.1 · pp. 19–21
  • Grid fusion derives obstacle-sensitive gradients from height data and maps them into a Dempster-Shafer sensor model, tuning thresholds to suppress small terrain variations while retaining curb-like gradients.The height grid is processed with Sobel operators, whose gradient peaks indicate obstacles; the mapping parameters distinguish fully drivable from undrivable gradients.Source: §4.1 · pp. 21–22

4.2 Vision

  • Four high-resolution cameras detect the vehicle’s lane and two adjacent lanes across a 175-degree field of view at up to 35 meters.The lane-detection output is processed by the artificial intelligence, while monocular color segmentation contributes to Grid Based Fusion.Source: §4.2 · p. 22
  • The system transforms synchronized camera images into a single top-view perspective, detects lane-marking features in HSV, and uses RANSAC to estimate lane-segment parameters robustly to outliers and artifacts.Four 1376x600 8-bit raw Bayer images are synchronously acquired at 14 frames per second.Source: §4.2 · pp. 23, 27
  • The area processor classifies terrain by comparing pixel colors with known drivable terrain, supporting situations without clear lane markings or high curbs.It marks lane markings and hard shadows as unknown drivability; it is needed for terrain that other sensors cannot distinguish, including off-road tracks.Source: §4.2 · pp. 28–29
  • Urban shadows, lane markings, and grass can mislead terrain classification, so preprocessing masks bright, dark, yellow, and vehicle-shadow pixels as unknown.The basic algorithm can treat large shadows as impassable and grass at street level as drivable; the preprocessing system uses bit masks to withhold unreliable pixel information.Source: §4.2 · pp. 30–32
  • Vision preprocessors flag problematic shadows, overexposed areas, lane markings, and the car’s own shadow as unknown, enabling lane changes despite the lane markings.The yellow preprocessor detects lane-marking colors across lighting conditions; the ego-shadow preprocessor identifies a low-brightness connected region in front of the vehicle.Source: §4.2 · pp. 33–36
  • The dynamic search polygon shifts according to the prior frame’s classified pixels, avoiding grass near the street edge that a static polygon would include as drivable.A bumper polygon moves within a boundary polygon; the movement is computed from the location and drivability of classified pixels inside the bumper polygon.Source: §4.2 · pp. 34, 35, 37

4.3 Artificial Intelligence

  • The DAMN architecture combines weighted curvature votes from waypoint, lane, obstacle, roadway, and zone behaviors to generate trajectory points and a drivable corridor.Behavior weights vary with conditions; the speed arbiter selects the lowest maximum speed proposed by its behaviors, and trajectory points extend far enough to stop at the current speed.Source: §4.3 · pp. 37–38
  • Interrupts extend DAMN to handle complex situations by reducing planned speed for a smooth stop, then suspending the arbiters until the interrupt returns control.Interrupts cover intersection yielding, queuing, overtaking, U-turns, blocked roads, parking, pause mode, and mission completion.Source: §4.3 · pp. 38–39

4.4 Vehicle Control

  • The longitudinal controller uses outer and inner loops to track speed: a P controller sets acceleration, while a PD controller tracks it through throttle and brake inputs.Measured drive-chain values yield a system lag of T = 0.6s; predefined logic prevents throttle and brake from activating simultaneously.Source: §4.4 · pp. 40–41
  • The lateral controller aims to follow a trajectory with minimal track error and smooth maneuvers, using a bicycle-model state-space representation that outputs track error.The model incorporates yaw rate, attitude angle, relative yaw angle, track error, and actual steering angle; its assumptions include zero longitudinal acceleration and constant wheel load.Source: §4.4 · pp. 42, 44, 45
  • Three degrees of freedom were controlled using steering-angle input, with feedback loops for track error and angle deviation plus curvature-based pilot control.The working point was 30 km/h, and the pilot control used bicycle-model parameters to calculate steering for the desired track.Source: §4.4 · p. 45
  • The lateral control strategy remained stable with quite low track error during all DARPA Urban Challenge tests and missions.Source: §4.4 · p. 46

4.5 Safety

  • Safety systems monitored hardware and software integrity, brought the car to a safe stop on error, and provided a remote E-stop plus independent module reset and restart.Source: §4.5 · pp. 46–47
  • A safety circuit linked the watchdog gateway, roof emergency buttons, and remote E-stop receiver to a relay that activated the parking brake when a system failed or was triggered.During emergency braking, the lateral controller could still hold the car on its course.Source: §4.5 · p. 48
  • The watchdog also improved overall reliability by monitoring non-safety subsystems; local daemons restarted failed processes, while a remote master watchdog detected computer freezes and initiated an emergency stop and powercycle.The CAN gateway initiated an emergency stop if the master watchdog failed.Source: §4.5 · pp. 48–49

5 System Development Process

CarOLO paired a staged software-development process with simulation and automated acceptance testing to develop and check Caroline’s complex software. The simulator provided a controlled setting for AI testing, while black-box tests supported repeated evaluation of complex traffic situations.

5 System Development Process

  • The five-step software workflow checked compilation, memory leaks, statement coverage, and runtime performance after executing test code.The final optimization step identifies time-consuming parts of an algorithm; the complete workflow runs on each checked-in revision.Source: §5 · p. 49, Figure 40 · p. 49
  • Interactive simulations of AI situations could be formalized as acceptance tests and run automatically after revisions.The team based requirements on DARPA documents and selected and prioritized iteration tasks through Scrum.Source: §5 · p. 50, §5.2 · p. 53

5.1 Simulator

  • The simulator let developers safely test AI functions without a real vehicle and supplied feedback by applying steering commands to the ego state and surroundings.For trajectory behavior, it reused the vehicle-control code and third-order B-splines to produce smoother motion considered sufficiently close to reality for intelligent-driving tests.Source: §5.1 · pp. 50, 52
  • The simulator represented world coordinates as object positions linked to objects and behaviors, with factories instantiating components and a control supervising the simulation.Motion behaviors were decoupled from objects, allowing behavior changes during a simulation and simplifying new behavior implementations.Source: §5.1 · pp. 50–52
  • Simulator adapters routed system calls to either the real system or simulator, while simulator-controlled time avoided confusing time-based algorithms with mixed time sources.Components were not to call system functions or retrieve time directly.Source: §5.1 · p. 53

5.2 Quality Assurance

  • Validators checked AI output against constraints such as object distances, lane departure, and speed limits after simulator steps, then summarized and reported test results.Their set covered basic requirements and restrictions for automatic checks of software revisions.Source: §5.2 · p. 53
  • Black-box acceptance tests enabled repeated testing of more complex traffic situations without requiring deep knowledge of the AI’s internal structure.The validator suite covered basic requirements and restrictions and could automatically check new software revisions.Source: §5.2 · pp. 53–54
  • Test drives used RNDFs and generated fusion-object and drivability data to model routes, obstacles, sensor noise, GPS drift, and terrain.Dynamic obstacles could follow individual RNDF routes, supporting intersection and more complex traffic scenarios.Source: §5.2 · pp. 54–56

6 The Race and Discussion

Caroline completed qualification tasks that included merging into traffic, parking, handling intersections, and rerouting around a barrier, qualifying for the final event. In the final race she drove 16.4 kilometers before being retired after a collision with MIT’s Talos, following earlier stops and an unresolved right-of-way incident.

6.1 National Qualification Event

  • In Area A, Caroline initially treated course barriers as stationary obstacles and drove into the opposite lane; later, surface-color differences led her toward the wrong route.The team modified parameters and behavior between attempts before demonstrating successful traffic merging.Source: §6.1 · pp. 57–59
  • In Area B, Caroline entered the free-navigation traffic circle smoothly, then parked between vehicles and left the zone to continue the course.The zone entrance was rough and had damaged other teams’ tires; the camera video showed heavy vibration as Caroline entered smoothly.Source: §6.1 · pp. 59–60
  • Caroline was one of only three vehicles to complete the National Qualification Event’s Area A course in time.She required several attempts to merge through moving traffic, and her successful final attempt demonstrated approximately eight perfect rounds.Source: §6.1 · pp. 57, 59, 60
  • In Area C, Caroline handled the intersections correctly, detected a barrier with no ground contact, rerouted by U-turn, and finished the course.The team had been unsure whether the sensors would detect the barrier because they could see through it.Source: §6.1 · pp. 60–61, Figure 52 · p. 61

6.2 Mandatory Practice for DARPA Urban Challenge Final Event

  • At final-event practice, Caroline twice failed to leave the start chute; the first failure involved parsing DARPA’s RNDF, and GPS jitter was later found to affect departure from the calculated trajectory.The team fixed the GPS issue and prepared Caroline for the following day’s event.Source: §6.2 · pp. 61–62

6.3 DARPA Urban Challenge Final Event

  • During the final race, Caroline became stuck while trying to pass a stopped vehicle after about 2.5 kilometers and needed human intervention to get free.She had interpreted the stopped vehicle as a stationary obstacle and attempted to pass it through a clearance beside the last car.Source: §6.3 · p. 62, Figure 54 · p. 63
  • After continuing, Caroline failed to yield to Talos at about 11.4 kilometers; the team could not determine why from its logs.The authors said sensor or AI interpretation problems could have contributed, but did not resolve the cause.Source: §6.3 · p. 63
  • Caroline drove 16.4 kilometers before DARPA retired her after a collision with MIT’s Talos; the team could not determine which car was responsible.The authors said further interpretation of the collision was impossible without detailed log files.Source: §6.3 · p. 64

7 Conclusion

Caroline autonomously performed urban missions at the DARPA Urban Challenge Final Event, driving approximately 17 kilometers in about three hours. Its design combined multi-sensor perception, real-time trajectory planning, supervisory safety monitoring, and a multi-level test process.

7 Conclusion

  • Team CarOLO’s Caroline is a standard 2006 Volkswagen Passat station wagon built to European specifications, able to detect and track stationary and dynamic obstacles at distances of up to 200 meters.Source: §7 · p. 66
  • CarOLO combined computer-science, mechanical-engineering, and electrical-engineering expertise, with substantial industrial sponsorship.New features and modifications went through unit tests or comprehensive simulations before vehicle deployment.Source: §7 · pp. 66–67
  • The system fused lidar, radar, and laser-scanner data, detected lanes and drivable areas, and planned real-time trajectories for urban driving.Its artificial intelligence handled situations such as intersection regulations and U-turns around blocked roads; a supervisory watchdog monitored hardware and software and triggered a safe stop on malfunctions.Source: §7 · p. 67
  • Approximately 17 kilometers in about three hours: Caroline drove autonomously in the DARPA Urban Challenge Final Event.The paper reports that she autonomously performed missions in urban environments.Source: §7 · p. 67
Loading 1409.6584v1…