PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Texture Manifold for Ground Terrain Recognition

Jia Xue, Hang Zhang, Kristin Dana

arXiv:1803.10896v2cs.CV

TL;DR

Ground terrain recognition needs to account for local spatial structure as well as texture details. DEP combines both representations, and DEP-manifold predicts texture coordinates to represent class relationships. DEP performs strongly across GTOS, MINC, and DTD, and reaches 82.18% multi-scale accuracy on GTOS-mobile.

  • Problem

    Ground terrain surfaces are not completely orderless, so local spatial order is an important recognition cue alongside texture details.

    Source: §1 · p. 2

  • Method

    DEP jointly processes orderless texture-encoding features and global-average-pooling features for local spatial information, while DEP-manifold predicts texture-image coordinates.

    Source: §1 · pp. 2–3

  • Results

    DEP shows excellent performance on GTOS, MINC, and DTD; on GTOS-mobile, multi-scale DEP accuracy is 82.18%, 9.02% better than ResNet and 6% better than Deep-TEN.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    DEP-manifold represents relationships between newly captured images and dataset images, providing a way to handle ambiguous class boundaries.

    Source: §1 · pp. 2–3

    The multi-scale preprocessing choice is empirically validated for resizing GTOS images to 256×256, 384×384, and 512×512.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

We present a texture network called Deep Encoding Pooling Network (DEP) for the task of ground terrain recognition. Recognition of ground terrain is an important task in establishing robot or vehicular control parameters, as well as for localization within an outdoor environment. The architecture of DEP integrates orderless texture details and local spatial information and the performance of DEP surpasses state-of-the-art methods for this task. The GTOS database (comprised of over 30,000 images of 40 classes of ground terrain in outdoor scenes) enables supervised recognition. For evaluation under realistic conditions, we use test images that are not from the existing GTOS dataset, but are instead from hand-held mobile phone videos of similar terrain. This new evaluation dataset, GTOS-mobile, consists of 81 videos of 31 classes of ground terrain such as grass, gravel, asphalt and sand. The resultant network shows excellent performance not only for GTOS-mobile, but also for more general databases (MINC and DTD). Leveraging the discriminant features learned from this network, we build a new texture manifold called DEP-manifold. We learn a parametric distribution in feature space in a fully supervised manner, which gives the distance relationship among classes and provides a means to implicitly represent ambiguous class boundaries. The source code and database are publicly available.

1. Introduction

Ground terrain recognition must account for local spatial structure as well as orderless texture details. DEP combines these cues, while DEP-manifold predicts texture-image coordinates in a learned manifold and GTOS-mobile evaluates recognition across different capture conditions.

1. Introduction

  • Local spatial order is an important recognition cue because terrain surfaces are not completely orderless.DEP is designed to balance ordered spatial information with an orderless representation.Source: §1 · p. 2
  • DEP jointly processes texture-encoding outputs that preserve appearance details and global-average-pooling outputs that preserve local spatial information.The two feature representations are processed with bilinear models.Source: §1 · p. 2
  • DEP shows excellent performance on GTOS and the more general MINC and DTD databases.The introduction reports results across all three databases without providing specific scores here.Source: §1 · p. 2
  • Ambiguous class boundaries motivate using manifold position to represent relationships between newly captured images and dataset images.The introduction gives asphalt/stone-asphalt and leaves/grass as examples of similar classes.Source: §1 · p. 2
  • DEP-manifold uses a neural network to predict texture-image coordinates in a manifold learned from DEP features.The method treats an embedding from non-parametric embedding algorithms as the target output.Source: §1 · p. 3
  • GTOS-mobile tests transfer from GTOS training images to handheld-phone videos captured with arbitrary lighting and viewpoints.The GTOS training set contains over 30,000 images; the introduction describes the mobile set as 81 videos of similar terrain classes.Source: §1 · p. 3

2. Related Work

Related work spans bilinear feature models, the growth of material-recognition datasets, and texture encoding within CNNs. The paper adapts these approaches to represent texture details and combine features.

2. Related Work

  • Bilinear CNNs use outer products of convolutional feature maps, but require matching map dimensions and compute outer products at each location.The paper instead applies bilinear models to fully connected-layer outputs, which are vectors without the same dimensionality restriction.Source: §2 · p. 3
  • Material-recognition datasets have grown from small lab collections to larger image sets collected in the wild.The cited overview contrasts roughly 100 images per class with datasets containing over 1,000 images per class.Source: §2 · p. 3
  • The paper uses a texture-encoding layer that brings dictionary learning and residual encoding into CNNs to capture orderless texture details.The layer is used as a component of the proposed network.Source: §2 · p. 3

3. Deep Encoding Pooling Network

DEP combines orderless texture encoding with spatial information from convolutional features, then models interactions between the two representations for classification.

3. Deep Encoding Pooling Network

  • The texture encoding layer globally pools descriptor-to-codeword residuals into codeword-specific vectors, with learnable assignment weights and smoothing factors.Increasing the number of learnable codewords can capture more texture details.Source: §3 · pp. 3–4
  • The bilinear model combines texture and spatial vectors through learnable weights, capturing pairwise correlations between their representations.Its two-factor form is linear in either factor when the other is held constant.Source: §3 · p. 4
  • DEP sends pretrained CNN features through texture-encoding and global-average-pooling branches, reduces both with fully connected layers, and bilinearly combines them for classification.In the ResNet-18 instantiation, the branches are reduced to 64 dimensions each before their 64 × 64 bilinear output is classified.Source: §3 · p. 4
  • Because the texture-encoding layer and bilinear models are differentiable, DEP parameters are trained end-to-end with stochastic gradient descent and back-propagation.Source: §3 · p. 4

4. Recognition Experiments

The experiments evaluate DEP on GTOS-mobile and on the broader DTD and MINC texture datasets. DEP combines texture encoding with spatial information, and the reported results favor it over the compared methods.

4. Recognition Experiments

  • DEP combines an encoding layer for texture appearance with global average pooling for spatial information, then applies bilinear models to the two representations.The experiments compare it with ResNet, Deep-TEN, and Bilinear-CNN using an identical training and evaluation procedure.Source: §4 · pp. 4–5

4.1. Dataset and Evaluation

  • GTOS-mobile tests 31 terrain classes using 6,066 frames from 81 handheld-phone videos, with varied viewpoints and lighting.Five GTOS classes were removed because materials were inaccessible, and asphalt and metal were merged.Source: §4.1 · pp. 5–6, Figure 4 · p. 5
  • Training uses 256×256 patches extracted from GTOS images resized at 256×256, 384×384, and 512×512 for multi-scale experiments.The paper motivates these scales as simulating changes in distance and viewing angle.Source: §4.1 · p. 6

4.2. Recognition Results

  • 82.18% accuracy on GTOS-mobile: multi-scale DEP is 9.02% better than ResNet and 6% better than Deep-TEN.Multi-scale training outperforms single-scale training for all approaches.Source: §4.2 · p. 6
  • DEP outperforms state-of-the-art algorithms on both DTD and MINC in single-scale experiments.The authors note that multi-scale training is likely to improve results for all methods.Source: §4.2 · p. 6, Table 3 · p. 6

5. Texture Manifold

DEP-manifold learns to predict 2D texture coordinates from DEP features. Its embedding distributes classes more evenly than DEP-parametric t-SNE, while test images can be projected without having been used to build the manifold.

Texture Manifold

  • A deep network predicts 2D manifold coordinates directly from 128-dimensional DEP features taken before the classification layer.The embedding network is trained from scratch, using 60,000 images from multi-scale GTOS.Source: §5 · p. 7
  • DEP-manifold gives a better-distributed 2D class embedding than DEP-parametric t-SNE, whose classes crowd in some regions and spread sparsely in others.Figure 7 compares embeddings of 60,000 images from multi-scale GTOS.Source: Figure 7 · p. 8, §5 · p. 8
  • Test images are embedded after the manifold is built, and cement images can remain within the cement margin despite imperfect recognition.The example places cement near the related stone-cement class.Source: §5 · p. 8
  • DEP recognition has few off-diagonal confusion-matrix entries, often near the diagonal, indicating errors to closely related classes.The matrix orders classes by a one-dimensional embedding of the 2D manifold.Source: §5 · p. 8

6. Conclusion

The conclusion identifies three contributions: DEP for terrain recognition, DEP-manifold for direct texture-coordinate prediction, and GTOS-mobile for testing transfer across image-capture methods.

6. Conclusion

  • DEP combines orderless representation with local spatial information for ground-terrain recognition.Source: §6 · p. 8
  • DEP-manifold uses a deep neural network to predict texture manifold coordinates directly.Source: §6 · p. 8
  • GTOS-mobile comprises 81 handheld-phone videos of terrain classes similar to GTOS, collected to evaluate transfer between capture methods within the same domain.Source: §6 · p. 8
Loading 1803.10896v2…