PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Neural Network-Based Image Representation for Visual Loop Closure Detection

Yi Hou, Hong Zhang, Shilin Zhou

arXiv:1504.05241v1cs.ROcs.CV

TL;DR

The paper asks whether pre-trained CNN representations can improve visual loop closure detection in SLAM, especially where hand-crafted descriptors face illumination and computational limitations. It evaluates descriptors from CNN layers against state-of-the-art hand-crafted alternatives and finds comparable performance without major lighting change, better performance under significant lighting change, and much faster extraction.

  • Problem

    Visual loop closure detection in SLAM needs image representations that remain effective under changing illumination, while existing hand-crafted descriptors have robustness and computational weaknesses.

  • Method

    The study extracts whole-image descriptors from every layer of a pre-trained CNN and compares selected CNN descriptors with state-of-the-art hand-crafted descriptors across lighting conditions.

  • Results

    CNN-based descriptors perform comparably without significant lighting change, outperform hand-crafted descriptors under significant lighting change, and extract one order faster on CPU and two orders faster on an entry-level GPU.

  • Takeaways & Limitations

    CNN-based image representations are a practical alternative to hand-crafted descriptors for visual loop closure detection, particularly in long-term environments with illumination variation.

Abstract

from arXiv · show

Deep convolutional neural networks (CNN) have recently been shown in many computer vision and pattern recog- nition applications to outperform by a significant margin state- of-the-art solutions that use traditional hand-crafted features. However, this impressive performance is yet to be fully exploited in robotics. In this paper, we focus one specific problem that can benefit from the recent development of the CNN technology, i.e., we focus on using a pre-trained CNN model as a method of generating an image representation appropriate for visual loop closure detection in SLAM (simultaneous localization and mapping). We perform a comprehensive evaluation of the outputs at the intermediate layers of a CNN as image descriptors, in comparison with state-of-the-art image descriptors, in terms of their ability to match images for detecting loop closures. The main conclusions of our study include: (a) CNN-based image representations perform comparably to state-of-the-art hand- crafted competitors in environments without significant lighting change, (b) they outperform state-of-the-art competitors when lighting changes significantly, and (c) they are also significantly faster to extract than the state-of-the-art hand-crafted features even on a conventional CPU and are two orders of magnitude faster on an entry-level GPU.

I. INTRODUCTION

The paper studies CNN-generated image representations for visual loop closure detection, targeting weaknesses of hand-crafted descriptors under illumination change and computational constraints. It evaluates CNN layers and compares selected descriptors with established alternatives.

  • Motivation: Visual loop closure detection determines whether a robot has returned to a previously visited location using captured images.The task supports consistent map construction by correcting accumulated errors.
  • Motivation: Image matching for loop closure consists of image description followed by similarity measurement, making the descriptor a critical component.The descriptor compresses an image into a compact, discriminating one-dimensional vector.
  • Motivation: Hand-crafted descriptors such as BoVW and GIST commonly suffer from limited illumination robustness and high extraction cost.These methods rely on human-designed feature engineering, although BoVW became a standard baseline in loop closure detection.
  • Research question: The study examines whether intermediate CNN layers provide better loop-closure representations than final fully connected layers, which lose spatial information.The paper notes that representations effective for classification or retrieval may not work well for visual loop closure detection.
  • Approach: The method uses a pre-trained Places CNN to generate whole-image descriptors from each layer, with deeper layers producing more abstract representations and varying descriptor lengths.The model was trained on more than 2.5 million images across 205 scene categories.
  • Contributions: CNN descriptors match hand-crafted descriptors without major illumination change, outperform them under lighting change, and extract faster.The reported speed advantage is one order of magnitude on CPU and two orders on an entry-level GPU.

II. RELATED WORK

Related work includes local-feature aggregation descriptors and whole-image descriptors for visual loop closure and retrieval. These approaches trade representation detail, robustness, memory footprint, and extraction cost.

  • Descriptor families: Hand-crafted descriptors are broadly divided into local-keypoint-based and entire-image-based representations.This distinction organizes the related visual descriptor families discussed in the paper.
  • Local descriptors: BoVW represents an image as a histogram of visual words formed by vector-quantizing local keypoint descriptors.It is described as a particularly successful descriptor in visual loop closure detection.
  • Local descriptors: Fisher vector uses a Gaussian mixture model whose means define cluster centers and whose covariances capture keypoint-descriptor distributions.The paper identifies FV as a leading hand-crafted image descriptor in some evaluations.
  • Local descriptors: VLAD simplifies Fisher vector by retaining first-order mean information, making it preferable when performance and memory footprint must be balanced.The paper reports that VLAD can be nearly comparable to FV in some cases.
  • Whole-image descriptors: Hand-crafted descriptors depend on local keypoint quality and can be expensive because keypoint detection and vector quantization are time-consuming.GIST was studied as a whole-image alternative to address these weaknesses.

B. Deep CNN-Based Features

Deep CNN features provide increasingly abstract visual representations and have achieved strong results in classification and retrieval, motivating their investigation for visual loop closure detection.

  • Deep CNN features: CNNs learn high-level features at increasing abstraction levels and have outperformed shallow hand-crafted features on major vision benchmarks.The related work highlights strong results in image classification and image retrieval.

III. CNN-BASED IMAGE DESCRIPTORS

The paper constructs whole-image descriptors from the outputs of a pre-trained CNN and examines how layer depth affects abstraction, spatial information, and descriptor suitability for loop closure detection.

  • Implementation: Caffe is used to extract CNN features from the standard multi-layer model.Descriptor dimensions vary according to the CNN layer and parameter settings.
  • CNN architecture: The CNN architecture contains five convolutional, three max-pooling, and three fully connected layers.Pooling follows the first, second, and fifth convolutional layers.
  • CNN architecture: Max-pooling provides translation invariance, reduces dimensionality, and builds local abstractions, whereas fully connected layers lose spatial information.The latter property may be undesirable for visual loop closure detection.
  • Layer representations: Deeper pooling layers are expected to retain substantial spatial information while providing richer semantic representations than shallower layers.Pool 5 is identified as a particularly promising layer for visual loop closure detection.
  • Descriptor construction: Each CNN layer output is treated as a feature vector and normalized, for example with the ℓ2-norm, to create a whole-image descriptor.The resulting descriptors are layer-by-layer abstract representations of the input image.

A. Datasets and Ground Truth Generation

The study evaluates loop-closure descriptors on established datasets with stable lighting and on UA Campus, created to test illumination variation. UA Campus revisits the same route across five times of day, with manually generated loop-closure ground truth.

  • City Centre and New College provide 1237 and 1073 image pairs, respectively, from outdoor urban robot runs with stable lighting and available loop-closure ground truth.
  • UA Campus covers about 650 meters and was recorded at five times of day to create illumination variation.
  • UA Campus images were manually matched to generate loop-closure ground truth across the changing-light sequences.

B. Algorithm Implementation Details

The implementation compares hand-crafted and CNN-based descriptors using established software and a scene-centric Places-CNN model. Descriptor performance is evaluated by nearest-neighbor matching and precision-recall measures across dataset-specific revisit experiments.

  • Hand-crafted baselines include BoVW, FV, VLAD, and GIST, while Caffe extracts descriptors from a reconstructed standard CNN architecture.
  • The Places-CNN model shares the standard ImageNet-CNN architecture but is trained on scene-centric data, making it better suited to scene recognition and loop closure detection.
  • Performance is evaluated using precision-recall curves and average precision after nearest-neighbor matching of each query image to the robot map by Euclidean distance.
  • UA Campus uses the 0620 sequence as the reference and treats four other times as revisits, producing four visual loop-closure experiments.

C. Comparison of CNN-Based Descriptors

CNN descriptors are compared across layers and against hand-crafted descriptors under stable and changing illumination. POOL5 is selected as the strongest compact representation, while CNN descriptors remain competitive with hand-crafted alternatives and are more robust to severe lighting changes.

  • POOL5 achieves the best result among all evaluated CNN layers.
  • FC6 performs best among the three fully connected layers, although fully connected representations are generally weaker than convolutional and pooling layers.
  • The last three convolutional layers have comparable performance, motivating CONV3 as their representative in later comparisons.
  • CNN-based descriptors perform similarly to hand-crafted descriptors on City Centre and New College, while remaining relatively insensitive to illumination changes on UA Campus.
  • Under dramatic lighting changes, convolutional and pooling layers significantly outperform fully connected layers.
  • POOL5 is considered the most appropriate descriptor when both performance and compactness matter, especially for large maps with many locations.

D. Comparison of Hand-Crafted and CNN Image Descriptors

CNN-based descriptors match hand-crafted descriptors under stable lighting, while showing greater robustness when illumination changes.

  • CNN-based and hand-crafted descriptors achieve similar performance on datasets without illumination change.The comparison uses three CNN descriptors and four hand-crafted features across three datasets.
  • CNN-based descriptors are relatively insensitive to lighting changes, whereas hand-crafted features are very sensitive.FC6 is an exception among CNN descriptors but still outperforms BoVW and GIST.
  • FC6 remains superior to BoVW and GIST under illumination change despite being the least robust CNN descriptor in that setting.

E. Computational Time

CNN descriptors are faster to extract than hand-crafted alternatives, with further acceleration on an entry-level GPU.

  • 0.155 seconds per image makes CPU-extracted CNN descriptors about 3 times faster than GIST and 10 times faster than other three state-of-the-art hand-crafted features.The average is computed over 649 images and excludes input-image and CNN-model loading time.
  • Average computational time per image is the reported comparison measure for the different feature descriptors.
  • 0.019 seconds per image makes GPU-extracted CNN descriptors approximately two orders of magnitude faster than state-of-the-art hand-crafted image descriptors.

V. CONCLUSION AND DISCUSSION

The study finds CNN descriptors competitive with hand-crafted descriptors under stable lighting, more effective under illumination change, and faster to extract, while identifying preliminary scope for further CNN exploration.

  • CNN-based descriptors perform similarly to hand-crafted descriptors without illumination change but outperform them by a significant margin under illumination change.
  • CNN-based descriptors are faster to extract by an order of magnitude on a CPU and two orders of magnitude on a low-cost, entry-level GPU.
  • POOL5 provides the best CNN descriptor choice in terms of detection accuracy and representation compactness.
  • The study remains preliminary because other online CNN models could be explored, alongside fine-tuning or training models specifically for loop closure detection.
Loading 1504.05241v1…