PaperlayerView exact revision on arXiv

Source-linked AI summary

Long-term Visual Localization using Semantically Segmented Images

Erik Stenborg, Carl Toft, Lars Hammarstrand

arXiv:1801.05269v2cs.CV

TL;DR

Long-term localization is challenged by seasonal and lighting changes that undermine visual feature matching, while maps should remain useful and compact. The paper uses semantically labeled 3-D point maps with semantically segmented images, and reports performance on par with the SIFT reference despite less informative map descriptors. The authors note that the problematic scenarios are improved but not fully solved.

  • Problem

    Seasonal and lighting changes can make feature matching between current images and older maps unreliable, while a useful environment representation should be compact and time-invariant.

    Source: §I · p. 1

  • Method

    The method localizes from semantically segmented images and a point map whose points are represented by 3-D position and semantic class rather than traditional descriptors.

    Source: §I · p. 2

  • Results

    Semantic localization performed on par with the reference algorithm despite using much less informative point descriptors in the map.

    Source: §VII · p. 6, §VI · p. 6

  • Takeaways & Limitations

    The results are promising for localization with less informative map descriptors, but the targeted problematic scenarios are improved rather than fully solved.

    Source: §VII · pp. 6–7

    Geometrically ambiguous scenes with a monotonous semantic class can make locations difficult to distinguish and cause the filter to favor positions where more map points are visible.

    Source: §VI · p. 6

Abstract

from arXiv · show

Robust cross-seasonal localization is one of the major challenges in long-term visual navigation of autonomous vehicles. In this paper, we exploit recent advances in semantic segmentation of images, i.e., where each pixel is assigned a label related to the type of object it represents, to attack the problem of long-term visual localization. We show that semantically labeled 3-D point maps of the environment, together with semantically segmented images, can be efficiently used for vehicle localization without the need for detailed feature descriptors (SIFT, SURF, etc.). Thus, instead of depending on hand-crafted feature descriptors, we rely on the training of an image segmenter. The resulting map takes up much less storage space compared to a traditional descriptor based map. A particle filter based semantic localization solution is compared to one based on SIFT-features, and even with large seasonal variations over the year we perform on par with the larger and more descriptive SIFT-features, and are able to localize with an error below 1 m most of the time.

I. INTRODUCTION

Long-term visual localization is difficult when lighting and seasonal changes disrupt feature matches, while a useful map should also remain compact. The paper proposes semantic point maps and segmented images for vehicle localization, evaluated against a SIFT-based method across data collected throughout a year.

I. INTRODUCTION

  • Seasonal and lighting changes can make traditional feature matching unreliable, causing poor positioning or localization failure as maps age.The paper frames the challenge as finding an environment description that is usable for localization, time-invariant, and compact.Source: §I · p. 1
  • The proposed method represents map points by 3-D position and semantic class, shifting seasonal invariance from feature descriptors to image segmentation.The paper compares its positioning performance with a SIFT point-feature map using data collected throughout a year.Source: §I · p. 2

II. PROBLEM STATEMENT

  • The localization task is to sequentially estimate the vehicle’s pose relative to a map from sensor observations.The observations are taken at irregular but known time intervals.Source: §II · p. 2, A. Observations · p. 2

1) Odometry:

The system combines noisy vehicle-state velocities, camera-derived image features, and a preconstructed map of 3-D landmarks with visibility information. The semantic and SIFT variants differ in how image features and map points are represented.

1) Odometry:

  • Odometry provides vehicle-frame linear and rotational velocities; measurements have additive Gaussian noise, with slowly varying bias on rotational velocity.Source: 1) Odometry · p. 2

2) Images:

  • Semantic-map image features are dense, with one element per pixel, whereas SIFT features are sparse detections paired with 128×1 descriptors.The cameras capture RGB images at about 15 Hz and 1024×768 resolution.Source: 2) Images · p. 2

B. Maps

  • Each map point has a global 3-D position, a descriptor, and visibility parameters that define its detection probability within a bounded wedge.The wedge is specified by two horizontal angles and a range from the point.Source: B. Maps · p. 2

C. Problem definition

The paper formulates localization as recursive estimation of the vehicle’s six-degree-of-freedom pose from observations and a map. Its process model propagates motion using speed measurements and preserves some probability density along the mapped road to support reacquisition.

C. Problem definition

  • The target is the posterior distribution of vehicle pose relative to the map, with state comprising global position and yaw, pitch, and roll.Source: C. Problem definition · p. 2

III. MODELS

  • A filtering solution combines a state-evolution process model with measurement models relating the state to observations.Source: §III · p. 2

A. Process model

  • The motion component uses speed measurements as input in a simple point-mass vehicle model.The process model is expressed using pose matrices, sample intervals, and motion noise.Source: A. Process model · p. 3
  • A mixture component projects a small fraction of density onto the mapped road, allowing a lost filter to reacquire its lateral position.The road is defined by the mapping vehicle’s route stored with the 3-D landmarks.Source: A. Process model · p. 3

B. Measurement model

The measurement model expresses image-feature likelihoods for both semantic and SIFT maps. Its formulation assumes known feature-to-map correspondences and conditional independence between paired measurements.

B. Measurement model

  • The measurement likelihood is defined from the current image’s feature points for both semantic and SIFT maps.Source: B. Measurement model · p. 3
  • The likelihood derivation assumes a data-association vector that identifies each image feature’s corresponding map feature, or marks it as absent.Source: B. Measurement model · p. 3
  • Given those associations and conditional independence of paired observations, the likelihood is factored using a model for each corresponding map point and image feature.The paper defers its treatment of unavailable data associations to Section IV.Source: B. Measurement model · p. 3

1) SIFT map:

The SIFT map models image-point projection error, while the semantic map uses dense pixel class labels and class likelihoods that account for unmapped pixels and possible occlusion.

1) SIFT map:

  • SIFT detections are modeled with zero-mean normally distributed projection error, using a pinhole camera projection with lens distortion.Camera mounting and intrinsic parameters are implicit in the projection function.Source: 1) SIFT map · p. 3

2) Semantic map:

  • Semantic image features are dense pixelwise class labels, matched to map points whose descriptors are scalar semantic classes.The classes include Cityscapes categories such as building and road.Source: 2) Semantic map · p. 3
  • Treating neighboring semantic pixels as independent can make measurements overconfident, increasingly so as their number grows.This motivates scaling the measurement likelihood and imposing a measurement cutoff.Source: 2) Semantic map · p. 3
  • For semantic measurements, the likelihood separates pixel coordinates from class descriptors; the coordinate factor is constant, leaving class probabilities to contribute.The class model distinguishes pixels with no projected map point from pixels with a projected point that may be occluded.Source: 2) Semantic map · pp. 3–4

A. SIFT filter

The SIFT reference filter uses a UKF and constructs feature-to-map associations by local descriptor matching followed by RANSAC culling.

A. SIFT filter

  • The reference SIFT localization uses a UKF with the motion and SIFT measurement models, rather than iterative optimization.RANSAC selects inliers instead of iterative reweighting; the on-road process component includes points visible from the nearest road point.Source: A. SIFT filter · p. 4
  • Candidate SIFT matches come from nearest-neighbor L2 descriptor matching in a pose-selected local map, filtered with Lowe’s ratio criterion.The local map is selected using the current pose estimate and map-point visibility.Source: A. SIFT filter · p. 4
  • A 3-point RANSAC step culls false correspondences by selecting the camera configuration with the most matches below a 6-pixel reprojection-error threshold.The method evaluates four camera views consistent with each sampled set of three correspondence pairs.Source: A. SIFT filter · p. 4

B. Semantic filter

The semantic filter recursively estimates vehicle pose with a bootstrap particle filter, projecting potentially visible map points into each particle’s image and tempering correlated measurement evidence.

B. Semantic filter

  • Semantic localization uses a bootstrap particle filter to recursively represent the posterior as weighted Dirac delta functions.The filter evaluates measurement likelihoods for individual particles.Source: B. Semantic filter · p. 4
  • Potentially visible map points are projected into the image to assign map points to pixels for each particle.Visibility may be approximated jointly for nearby particles using their mean position and map visibility parameters.Source: B. Semantic filter · p. 4
  • Because conditional independence overstates evidence from correlated measurements, the filter raises the measurement likelihood to a power below 1.The scaling parameter is 3, and contributions stop increasing beyond 400 projected map points.Source: B. Semantic filter · p. 5

V. EVALUATION

Evaluation uses CMU route sequences spanning seasonal, weather, and lighting changes; the supplied figures illustrate semantic map projection and per-point storage requirements.

V. EVALUATION

  • Twelve of 16 CMU dataset runs cover a roughly nine-kilometer Pittsburgh route recorded from September 2010 to September 2011.The selected runs capture seasonal, weather, and lighting variation across urban, suburban, and park settings.Source: §V · p. 5
  • In Fig. 3, projected map points overlay a segmented image; mapping and localization are two weeks apart in the example.Black dots are mainly building, green denotes vegetation, and yellow denotes pole-class points.Source: Figure 3 · p. 5
  • Fig. 4 compares per-point storage requirements, with 3-D position and visibility cone required regardless of descriptor.Descriptors are quantized to 8-bit resolution; semantic class descriptors may fit in one or two bytes when points are not at corners where three or more classes meet.Source: Figure 4 · p. 5

A. Map creation

The map is built from the first sequence using structure from motion, then assigns each 3-D point SIFT or semantic descriptors. A separate sequence-alignment procedure provides more reliable ground truth for evaluation.

A. Map creation

  • The map uses the first sequence from September 1, 2010; structure from motion with GPS and odometry constraints initializes the trajectory, followed by bundle adjustment.The remaining sequences are reserved for localization evaluation; low-speed and stopped images are culled, and the sequence is split into smaller mapping parts because of limited computing resources.Source: A. Map creation · p. 5
  • Semantic point descriptors use normalized class histograms from 7×7 pixel neighborhoods; SIFT point descriptors average the descriptors observed across views.The marginal class distribution is estimated from pixels in the mapping sequence, with a manual adjustment that raises dynamic-object probabilities and lowers stationary-object probabilities.Source: A. Map creation · p. 5
  • SIFT descriptors take more than six times the storage of even the most naive semantic class descriptor, while storing only the top three classes reduces a point descriptor to 39 bits.The semantic point cloud is colored by its most likely category in Fig. 5.Source: A. Map creation · pp. 5–6, Figure 5 · p. 6

B. Ground truth

  • Ground truth is refined by manually matching sequences and optimizing poses with the structure-from-motion pipeline; odometry is then derived from relative pose motion with added noise and bias.Additional Gaussian noise has variance 2.5 · 10^-5 for angular velocity and 4 · 10^-4 for the velocity vector.Source: B. Ground truth · p. 6

VI. RESULTS

Across the evaluation, semantic localization performed on par with the SIFT reference, but the results did not strongly support increased long-term robustness. The discussion identifies classification errors and geometrically ambiguous scenes as failure sources, and says the targeted scenarios were improved but not fully solved.

VI. RESULTS

  • Semantic localization can fail when map-point and image classes mismatch because of misclassification, with terrain beside roads often labeled road or vegetation, especially in snow.Fig. 7 shows an image and its segmentation at a semantic-localization failure site.Source: §VI · p. 6, Figure 7 · p. 7
  • Semantic localization performed on par with the SIFT reference across the evaluation; Fig. 6 shows distance-error histograms for 11 trials and three environment types.The histograms compare the fraction of samples within 0.5 m, 1 m, and 2 m; taller bars indicate more samples localized within a given error limit.Source: Figure 6 · p. 6, §VI · p. 6
  • Monotonous roadside classes such as a high wall or tall forest can make locations look alike, while the filter may favor positions where textured map points remain visible longer.This geometric ambiguity can draw the estimate toward a high-texture area even when the surrounding scene is otherwise uniform.Source: §VI · p. 6
  • SIFT localization typically fails when too few image-map correspondences survive RANSAC, allowing dead-reckoning error to accumulate; this occurred mostly in the vegetation-dominated Park area.Source: §VI · p. 6

VII. DISCUSSION

  • The results do not strongly support increased long-term robustness, and the targeted scenarios are improved but not fully solved without segmentation changes.The authors suggest broader environmental training data, including winter and extreme lighting, and classes such as road markings or separate vegetation trunk and foliage labels.Source: §VII · pp. 6–7
Loading 1801.05269v2…