PaperlayerView exact revision on arXiv

Source-linked AI summary

Find Your Own Way: Weakly-Supervised Segmentation of Path Proposals for Urban Autonomy

Dan Barnes, Will Maddern, Ingmar Posner

arXiv:1610.01238v3cs.ROcs.AIcs.CVcs.LG

TL;DR

Urban autonomous driving depends on road-scene understanding, and this paper addresses monocular segmentation of proposed paths. It generates training labels from recorded routes and vehicle sensors, then reports path and obstacle segmentation across varied conditions, with sensor-suite limitations in some cases.

  • Problem

    How to segment a driver's proposed drivable path from a monocular image in urban scenes, where road-scene understanding is critical to safe vehicle decisions.

    Source: §I · p. 1

  • Method

    The method uses recorded routes, driver behaviour, and additional vehicle sensors to generate labelled images without manual annotation and train a semantic segmentation network.

    Source: §VI · p. 7

  • Results

    Mean IoU scores exceeded 80% in all reported Oxford lighting and weather conditions, and the approach demonstrated reliable path and obstacle segmentation across varied conditions.

    Source: A. Oxford Dataset · p. 5, §VI · p. 7

  • Takeaways & Limitations

    At intersections, the network frequently labels multiple possible paths despite training labels containing only one proposed path per image; route generalisation is shown qualitatively.

    Source: D. Route Generalisation · p. 7

    Some conditions yield unusable path segmentations, mostly due to poor exposure or limited field of view in the sensor suite; higher-quality or more numerous cameras could address these failures.

    Source: C. Limitations · pp. 6–7

Abstract

from arXiv · show

We present a weakly-supervised approach to segmenting proposed drivable paths in images with the goal of autonomous driving in complex urban environments. Using recorded routes from a data collection vehicle, our proposed method generates vast quantities of labelled images containing proposed paths and obstacles without requiring manual annotation, which we then use to train a deep semantic segmentation network. With the trained network we can segment proposed paths and obstacles at run-time using a vehicle equipped with only a monocular camera without relying on explicit modelling of road or lane markings. We evaluate our method on the large-scale KITTI and Oxford RobotCar datasets and demonstrate reliable path proposal and obstacle segmentation in a wide variety of environments under a range of lighting, weather and traffic conditions. We illustrate how the method can generalise to multiple path proposals at intersections and outline plans to incorporate the system into a framework for autonomous urban driving.

I. INTRODUCTION

The paper targets path segmentation for urban driving without dependence on explicit lane models or markings. It generates training labels from recorded routes and vehicle sensors, then evaluates the approach on KITTI and Oxford RobotCar.

I. INTRODUCTION

  • Lane-marking-dependent systems are suited to simple, clearly marked roads, motivating an alternative for complex urban, rural, and undeveloped settings.Current commercial systems typically rely on explicit lane definitions and road markings.Source: §I · p. 1
  • The method segments driver-expected path proposals from monocular images without explicitly modelling lanes or lane markings.A path proposal is defined as a route a driver would be expected to take through a particular road and traffic configuration.Source: §I · p. 1
  • Recorded routes and vehicle sensors generate large quantities of labelled training images without manual annotation, spanning varied road, traffic, lighting, and weather conditions.The labels train a deep semantic segmentation network for monocular-image path proposal segmentation.Source: §I · p. 1
  • The evaluation uses KITTI and Oxford RobotCar data, including KITTI Road, Object, and Tracking benchmarks and Oxford lighting and weather conditions.Oxford RobotCar comprises over 1000 km of recorded driving in Oxford over a year.Source: §I · pp. 1–2

II. RELATED WORK

Prior approaches estimate drivable paths using road features, lane models, or manually labelled segmentation data, while synthetic and direct-behaviour methods have distinct constraints. The paper transitions to generating weakly supervised path-segmentation training data from recorded vehicle runs.

II. RELATED WORK

  • Traditional camera-based methods extract road and lane features, fit models, and fuse hypotheses over time, but can struggle with occlusions, shadows, changing light, unstructured roads, and missing markings.Radar or LIDAR can increase robustness at added sensor cost.Source: §II · p. 2
  • Deep semantic segmentation uses full-image context and is more robust than earlier feature-based methods, but large-scale manually labelled road-scene datasets are costly to produce.CamVid and Cityscapes contain 700 and 5,000 labelled frames, respectively.Source: §II · p. 2
  • Synthetic-image methods offer large-scale labels, but rendered images may not reflect real scenes and simulated traffic may not resemble real traffic; 3D label transfer requires sophisticated reconstructions and manual 3D annotations.Source: §II · p. 2
  • Direct image-to-behaviour methods use driver-generated labels, but their generalisation to multiple possible paths at intersections is unclear; this approach labels proposed paths while leaving path choice to a planner.Source: §II · p. 2

III. WEAKLY-SUPERVISED SEGMENTATION

  • The next section introduces weakly supervised training-data generation for path segmentation from video and sensor data recorded from a manually driven vehicle.Source: §III · p. 2

A. Sensor Configuration

Training-label collection uses vehicle odometry and obstacle sensing alongside camera images, although the trained network needs only monocular input. Labels are formed by projecting the driver's future path into images and accounting for detected obstacles.

A. Sensor Configuration

  • The data-collection vehicle needs motion estimates and 3D obstacle detection to prevent dynamic objects from being labelled as drivable; stereo odometry and LIDAR are used, though alternatives are possible.The trained network requires only a monocular image at inference time.Source: A. Sensor Configuration · p. 2

B. Weakly-Supervised Labelling

  • Weak labels combine the driver's future trajectory projected into each image with LIDAR-detected object labels.The method assumes the proposed path corresponds to the route chosen by the data-collection driver.Source: B. Weakly-Supervised Labelling · p. 2

1) Proposed path projection:

  • Path projection transforms calibrated wheel-ground contact points through vehicle-odometry poses, then fills image-space quadrilaterals across sequential future frames.The projected path exceeds 60 metres; higher camera viewpoints may require greater distance, while accumulated odometry error affects far-field projections.Source: 1) Proposed path projection · p. 3

2) Obstacle projection:

The method projects the vehicle’s future path and LIDAR obstacle points into the image, assigning obstacle labels precedence so vehicles are not mislabeled as drivable. Pixels labelled as neither path nor obstacle remain unknown for higher-level planning.

  • Without obstacle labels, projecting the driven path can mark vehicles in the same lane as drivable; LIDAR obstacle labels prevent this error.Obstacle labels take precedence over proposed-path labels.Source: 2) Obstacle projection · p. 3
  • Obstacle points are projected from the LIDAR scan into the camera image, and pixels on and above each point are labelled non-drivable.This also labels locations above and behind detected obstacles.Source: 2) Obstacle projection · p. 3
  • Projected left and right wheel contact points define path labels by filling quadrilaterals between successive points along the vehicle’s trajectory.The path contacts are projected up to k frames ahead into the current image.Source: Figure 3 · p. 3
  • Locations labelled as neither path nor obstacle are treated as unknown because whether to enter them is left to a higher-level planner.Examples include road outside the current lane, kerbs, pavements, and ditches.Source: 2) Obstacle projection · p. 3

C. Semantic Segmentation

The system trains a semantic segmentation network on automatically generated path and obstacle labels, with training-data sampling designed to represent different turning angles. Obstacle labels address path projections that would otherwise mark traffic as drivable.

  • SegNet classifies automatically generated proposed-path, obstacle, and unknown-area labels from monocular images, while retaining real-time evaluation for vehicle deployment.The labels are generated from recorded images before training.Source: C. Semantic Segmentation · p. 3
  • Because recorded routes are biased toward straight driving, frames are subsampled by turning angle using a histogram of average yaw rates.The data are first subsampled to 4 Hz, then sampled from yaw-rate histogram bins.Source: C. Semantic Segmentation · pp. 3–4
  • Obstacle labels mark vehicles, cyclists, pedestrians, and static occluders as non-drivable where the projected path alone could label them as drivable.The figure contrasts path-only labels with labels after LIDAR obstacles are added.Source: Figure 4 · p. 4

IV. EXPERIMENTAL SETUP

The experimental setup uses separate models trained on KITTI Raw and Oxford RobotCar data, collected with vehicles that have different sensor setups.

  • The authors build one evaluation model using KITTI Raw and another using Oxford RobotCar.Source: §IV · p. 4
  • The two datasets were collected using different vehicles with different sensor setups.Source: §IV · p. 4
  • The vehicle and sensor configurations for the two datasets are summarized in Table I.Source: §IV · p. 4

A. Platform Specifications

The study collects weakly supervised labels with vehicle-mounted sensors and trains SegNet on images from KITTI and Oxford RobotCar. The setup spans varied routes and conditions, with separate training and validation data and checks against evaluation-image overlap.

A. Platform Specifications

  • SegNet uses an encoder-decoder architecture that compresses feature representations spatially before producing full-resolution per-pixel class predictions.Source: Figure 5 · p. 4
  • The KITTI Velodyne setup labels points more than 0.25 m above a MLESAC-fitted ground plane as obstacles, while Oxford uses filtered LIDAR object contours.The figure illustrates the KITTI ground-plane filtering and resulting image labels.Source: A. Platform Specifications · p. 4, Figure 6 · p. 5

B. Network Training

  • Training uses 24,443 KITTI images and 49,980 Oxford images, with Oxford traversals spanning overcast, sun, rain, night, and snow conditions.The weakly supervised labels require no manual annotation and cover diverse KITTI locations and Oxford conditions.Source: B. Network Training · p. 4, Figure 7 · p. 5
  • Both datasets use the standard SegNet architecture, trained for 100 epochs after a random 75% training and 25% validation split.The best-performing model is selected using validation results.Source: B. Network Training · p. 4
  • For KITTI Road comparison, a separate SegNet is trained on the benchmark’s provided training images; KITTI object and tracking evaluation images do not overlap with weak-label training images.Source: B. Network Training · p. 5

V. RESULTS

The Oxford model was tested across sunny, cloudy, night, and rainy conditions, achieving mean IoU above 80% in each. It also segmented paths and obstacles across changing conditions and inferred routes where road or lane markings were absent.

A. Oxford Dataset

  • At the same location, the network correctly segmented proposed paths and obstacles across sunny, rainy, snowy, and night-time appearances.The figure includes cyclists, other vehicles, and road barriers among the labelled obstacles.Source: Figure 8 · p. 5, A. Oxford Dataset · p. 5
  • Mean IoU scores exceeded 80% in all tested conditions, with highest performance in cloudy weather and lowest at night due to reduced image quality in low-light conditions.Source: A. Oxford Dataset · p. 5
  • The network inferred valid paths from road-scene context at locations without explicit road or lane markings.The paper also reports an example on a gravel road not seen during training.Source: A. Oxford Dataset · p. 5

B. KITTI Benchmarks

On KITTI ego-lane segmentation, the weakly supervised model outperformed a model trained on the provided manual labels, despite using no manual annotations or explicit lane-marking encoding. The benchmark sections use alternative metrics because KITTI tasks do not exactly match the network’s path and obstacle labels.

B. KITTI Benchmarks:

  • The authors present alternative KITTI metrics because neither ego-lane segmentation nor object detection exactly matches the network’s outputs.The same model produces both path and obstacle labels in each test image.Source: B. KITTI Benchmarks · pp. 5–6

1) Ego-lane Segmentation:

  • The weakly supervised model achieved a 20% increase in max F-score and a 15% increase in precision, exceeding 90% in total, versus the model trained on manually annotated images.The authors note that ego-lane and proposed-path definitions differ, so this comparison is not competitive with more sophisticated KITTI leaderboard architectures.Source: 1) Ego-lane Segmentation · p. 6
  • KITTI ego-lane is the closest benchmark analogue to a proposed path, but its benchmark definition differs from the network’s proposed-path segmentation.The KITTI ego-lane set contains 95 training and 96 test images with manually annotated labels.Source: B. KITTI Benchmarks · p. 5, 1) Ego-lane Segmentation · p. 6
  • The authors suggest weakly supervised training could serve as pre-training to improve results even when manually annotated data is available.They contrast this with manually annotated datasets, which they describe as more time-consuming and expensive to produce.Source: 1) Ego-lane Segmentation · p. 6

2) Object Detection:

On KITTI Object and Tracking data, the approach labels benchmark objects as obstacles across classes, with especially strong instance recall at a 50% obstacle-pixel threshold. The paper also notes sensor-related path failures and shows that manual-label training performed poorly compared with weakly supervised training.

2) Object Detection:

  • The weakly supervised approach reliably labelled KITTI objects as obstacles across classes, performing especially well at the 50% instance-recall threshold.The evaluation covers 15,047 images and 87,343 object instances, using pixel recall and instance-recall thresholds of 50% and 75%.Source: 2) Object Detection · p. 6, Figure 11 · p. 7, Table V · p. 7
  • The KITTI evaluation counts an object instance as detected when a specified fraction of its bounding-box pixels are labelled as obstacles, despite the benchmark and method defining objects differently.The figure uses a 75% threshold and notes that tight obstacle outlines can leave parts of a bounding box, such as vehicle undercarriages, unlabelled.Source: 2) Object Detection · p. 6, Figure 11 · p. 7

C. Limitations

  • A SegNet trained on the small manually annotated KITTI Road set performed poorly compared with one trained on the larger weakly supervised dataset.The figure presents example ego-lane segmentation outputs for the two models.Source: Figure 10 · p. 7
  • Poor exposure and limited camera field of view are identified as causes of path-segmentation failures.The paper proposes additional, higher-quality cameras as a possible remedy.Source: C. Limitations · pp. 6–7

D. Route Generalisation

Although training labels provide only one path per image, the network often labels multiple possible paths at test-time intersections. The paper presents this route generalisation qualitatively, without ground truth to evaluate it, and concludes that the approach works across varied conditions while identifying sensor-related failure cases.

D. Route Generalisation

  • The network frequently labelled multiple possible paths at test-time intersections, although training labels contain only the data-collection trajectory’s single path.The authors present this route generalisation qualitatively because no ground truth is currently available to evaluate it.Source: D. Route Generalisation · p. 7

VI. CONCLUSIONS

  • The approach uses recorded driving behaviour and multiple sensors to generate large quantities of labelled training data without manual image annotation.The trained network uses a monocular camera and does not rely on explicit road markings or lane models.Source: §VI · p. 7
  • Overexposure or underexposure can cause incorrect path segmentation, while tight turns may place the path outside the camera’s field of view.The paper suggests a high-dynamic-range camera for exposure issues and a wider-view lens or surround cameras for field-of-view limits.Source: Figure 12 · p. 7
  • The authors report ego-lane performance above manually trained networks, reliable obstacle detections, and path segmentation across varied lighting, weather, and traffic conditions.They plan to integrate the network with a planning framework for autonomous driving in complex urban environments.Source: §VI · p. 7
Loading 1610.01238v3…