PaperlayerView exact revision on arXiv

Source-linked AI summary

Place Categorization and Semantic Mapping on a Mobile Robot

Niko Sünderhauf, Feras Dayoub, Sean McMahon, Ben Talbot, Ruth Schulz, Peter Corke, Gordon Wyeth, Ben Upcroft, Michael Milford

arXiv:1507.02428v1cs.RO

TL;DR

Robots need semantic place labels for decisions beyond navigation, but standard ConvNets cannot recognize classes absent from training. The paper adds online one-vs-all classifiers and Bayesian filtering to a ConvNet-based mapping system; it reports benefits for object recognition and navigation-related behavior.

  • Problem

    Robots need semantic place information for higher-level decisions, while closed-set classifiers cannot recognize categories outside their fixed training classes.

    Source: §I · p. 1, C. The Closed-Set Limitation · pp. 2–3

  • Method

    The system augments a ConvNet with online one-vs-all classifiers and embeds classification in a Bayesian filter before building a map from place labels.

    Source: §I · pp. 1–2

  • Results

    Semantic place information supports robotic object detection and recognition, and semantic maps can influence robot navigation behavior.

    Source: §VII · pp. 7–8

  • Takeaways & Limitations

    Semantic information can support advanced robotic tasks, especially human-robot collaboration, where people describe places, goals, and objects using semantic categories.

    Source: §VII · p. 7

    A classifier trained on fixed classes cannot recognize novel categories at test time, a constraint the paper says is unrealistic for long-term robotics.

    Source: C. The Closed-Set Limitation · pp. 2–3

Abstract

from arXiv · show

In this paper we focus on the challenging problem of place categorization and semantic mapping on a robot without environment-specific training. Motivated by their ongoing success in various visual recognition tasks, we build our system upon a state-of-the-art convolutional network. We overcome its closed-set limitations by complementing the network with a series of one-vs-all classifiers that can learn to recognize new semantic classes online. Prior domain knowledge is incorporated by embedding the classification system into a Bayesian filter framework that also ensures temporal coherence. We evaluate the classification accuracy of the system on a robot that maps a variety of places on our campus in real-time. We show how semantic information can boost robotic object detection performance and how the semantic map can be used to modulate the robot's behaviour during navigation tasks. The system is made available to the community as a ROS module.

I. INTRODUCTION

The paper targets transferable place categorization and semantic mapping without environment-specific training, so robots can identify what kind of place they are in and use that information for higher-level decisions. It combines a ConvNet with online class expansion and temporal filtering to address the network’s closed-set constraint.

I. INTRODUCTION

  • Place categorization assigns semantic labels to places, while semantic mapping combines those labels with a map to support decisions beyond localization and obstacle avoidance.The motivation includes high-level decision processes, robot-human interaction, and modulating robot behaviour.Source: §I · p. 1
  • The system is designed to transfer across environments without environment-specific training, unlike methods that require environment-specific training or fine-tuning.The paper identifies this as a distinction from prior semantic mapping systems in robotics.Source: §I · p. 1
  • One-vs-all classifiers add online recognition of new classes to the ConvNet, while a Bayesian filter uses temporal coherence and prior knowledge to reduce spurious classifications.The system is evaluated in real time across indoor and outdoor places, and its semantic information is also used for object recognition and navigation tasks.Source: §I · pp. 1–2

II. RELATED WORK

  • The paper builds on recent ConvNet progress in place categorization, including a network reported to improve performance on the SUN benchmark.The cited work used a ConvNet trained for place categorization.Source: §II · p. 2

A. Semantic Mapping in Robotics

Prior robotic semantic mapping systems used varied sensing and mapping approaches, but often had limited evidence of transferability across environments. The paper situates learned ConvNet features as an alternative to hand-crafted features and identifies the closed-set assumption as a poor fit for long-term robotics.

A. Semantic Mapping in Robotics

  • Earlier robotic mapping work combined vision, laser sensing, object cues, or human input, but evaluations often used visually similar settings or lacked publicly available data for comparison.Other place-discovery work clustered sensor data over time without assigning semantic labels.Source: A. Semantic Mapping in Robotics · p. 2

B. Features for Semantic Classification

  • Earlier vision-based systems relied on fixed hand-crafted image features, while ConvNets learn features and had recently outperformed competing approaches on semantic place categorization.The passage describes ConvNets as part of a broader shift in visual recognition away from hand-crafted features.Source: B. Features for Semantic Classification · p. 2

C. The Closed-Set Limitation

  • Closed-set classifiers assume test images belong to classes seen during training, an assumption the paper says is unrealistic for long-term robotic operation.The paper notes that overcoming this limitation remained an active research area without a widely accepted solution.Source: C. The Closed-Set Limitation · pp. 2–3

D. Summary

The paper presents a transferable semantic-mapping system that combines ConvNet classification, online class expansion, Bayesian temporal filtering, and mapping. Its Places205-based classifier is intended to generalize to new environments without retraining or fine-tuning.

D. Summary

  • Prior robotic systems often lacked clear separation between training and testing data, and the paper identifies its combination of ConvNets, online one-vs-all learning, and Bayesian temporal filtering as unexplored for vision-based semantic mapping.The authors link weak train-test separation to questionable transferability and generalizability.Source: D. Summary · p. 3

III. SYSTEM OVERVIEW

  • The system has four parts: a per-image ConvNet, one-vs-all classifiers for unseen classes, a Bayesian filter for temporal coherence, and a mapping subsystem that accumulates place labels.The complete system is provided as a ROS module.Source: §III · p. 3

A. Transferable Place Categorization

  • The Places205 network was trained on 2.5 million images across 205 semantic categories and is described as generalizing to unseen deployment environments without retraining or fine-tuning.An earlier AlexNet-based system scored 46.2% top-1 accuracy on SUN-397, compared with 47.2% for; Places205 achieved 54.3%, prompting the switch.Source: A. Transferable Place Categorization · p. 3

B. Expandable Place Categorization

The paper addresses closed-set classification by extending a Places205 ConvNet with one-vs-all classifiers for new classes. The combined classification output assumes independence among the original and added class labels.

B. Expandable Place Categorization

  • Robotics requires recognizing scene categories beyond the classes known during training, which closed-set classifiers do not accommodate.The paper notes that retraining the ConvNet can take days and require hundreds or thousands of new-class images.Source: B. Expandable Place Categorization · p. 3
  • Random Forest one-vs-all classifiers use Places205 fc7 features to add new classes without retraining the ConvNet.The paper says they can be trained in under a minute with roughly 10–100 examples.Source: B. Expandable Place Categorization · p. 3
  • The combined likelihood joins probabilities from the ConvNet and added-class classifiers, assuming independence between their labels and among added classes.The combined vector contains the original labels x0…xn and added labels y0…ym.Source: B. Expandable Place Categorization · p. 4

C. Bayesian Filtering over Class Labels for Temporal Coherence

The paper estimates place labels from a temporally coherent image stream with a Bayesian filter. It also incorporates prior knowledge about which categories are plausible in the robot’s environment.

C. Bayesian Filtering over Class Labels for Temporal Coherence

  • The Bayesian filter estimates a distribution over place labels from current and past images under a first-order Markov assumption.The current combined likelihood is L(It|x̂t).Source: C. Bayesian Filtering over Class… · p. 4

D. Incorporating prior knowledge

  • An additional prior term can down-weight Places205 categories that are unlikely or impossible in the robot’s deployment environment.The paper denotes this prior knowledge as p(x̂).Source: D. Incorporating prior knowledge · p. 4

E. Semantic mapping

The semantic mapping component combines place-class probabilities with laser scans to build a layered map. Each semantic layer tracks the probability that cells belong to its category.

E. Semantic mapping

  • The map maintains one layer per semantic category, where cells encode category probabilities rather than occupancy.Level zero remains a regular occupancy grid.Source: E. Semantic mapping · p. 4, Figure 2 · p. 4
  • Class probabilities are propagated along camera-visible laser rays to update map cells with a recursive Bayes filter.Unoccupied cells within 5 meters of the robot and in the camera’s field of view receive the currently observed semantic label.Source: E. Semantic mapping · p. 4
  • Log-odds updates let cell probabilities adapt to later observations, so spurious classifications do not permanently corrupt the map.The update is clamped between lower and upper bounds.Source: E. Semantic mapping · p. 4
  • Different traversal costs for semantic labels can guide path planning, such as avoiding busy places during certain times.The paper gives the food court at lunchtime as an example.Source: E. Semantic mapping · p. 4

IV. EXPERIMENTS, EVALUATION AND RESULTS

The system was evaluated on campus images from three robot-mounted cameras across nine environments. RGB performed best, grayscale was close on average, and Ladybug performed worst.

A. Place Categorization on a Real Robot

B. The Effect of the Bayes Filter

The Bayes filter makes place classifications more stable by smoothing spurious results and incorporating prior knowledge. Campus maps show mixed place categories, with image-level classification percentages that do not necessarily describe correctly labeled map area.

B. The Effect of the Bayes Filter

  • The Bayes filter produces a more stable maximum-a-posteriori solution than maximum likelihood alone, smoothing spurious results and reducing false classifications.Without prior domain knowledge, the maximum-likelihood solution uses a uniform prior over all 205 class labels.Source: B. The Effect of the Bayes Filter · p. 5

C. Semantic Mapping Results

  • The campus maps render only winning labels and show mixed categories within places, including offices, corridors, and a kitchen in one office environment.The percentages refer to correctly classified images, not necessarily correctly labeled map area.Source: C. Semantic Mapping Results · p. 5, Figure 5 · p. 6
  • The supermarket map includes a clothing-store area confirmed by manual inspection, but windows facing campus led to incoherent botanical-garden and parking-lot classifications.In the food court, the system labeled table-and-chair areas as restaurant, food stalls as food court, and transition areas as lobby.Source: C. Semantic Mapping Results · p. 6

D. Expanding the Classifier with New Classes

The authors extended the classifier with an online-learned door class, a category unavailable in the dataset. It detected most doors in an office test, with false detections and misses under challenging conditions.

D. Expanding the Classifier with New Classes

  • The classifier was trained with 80 positive elevator-door examples and 26,000 negatives, although the dataset had no proper door category.The positive examples came from the elevator-door3 category of SUN-397.Source: D. Expanding the Classifier with New… · p. 6
  • On 1,332 office images, the door classifier detected 8 of 10 doors and produced two false positives.The two missed doors were visible for only four frames or were at the end of a long unlit corridor; one false positive was a wooden plank.Source: D. Expanding the Classifier with New… · p. 6
  • Training the one-vs-all classifier took 57 seconds on a standard desktop machine.The implementation used Python’s scikit-learn.Source: D. Expanding the Classifier with New… · p. 6

A. Semantic Place Information Boosts Object Detection

The paper uses semantic place labels as context for object recognition and as costs for path planning. In a kitchen experiment, combining place information with object classification improved top-1 and top-5 accuracy; place-dependent costs support different routing preferences.

A. Semantic Place Information Boosts Object Detection

  • Object-class priors conditioned on the maximum-a-posteriori scene category were learned from NYU2 object labels, assuming independence between the image patch and scene category.Nonoccurring classes and classes absent from either NYU2 or ImageNet received a small, non-zero default prior.Source: A. Semantic Place Information Boosts… · p. 6
  • Fusing semantic place information with object detection increased top-1 accuracy from 0% to 54% and top-5 accuracy from 15% to 100% in a kitchen experiment.Fig. 6 contrasts original AlexNet top-5 results with results using the semantic scene class to condition object-class priors.Source: A. Semantic Place Information Boosts… · p. 6, Figure 6 · p. 7

B. Path Planning on a Semantic Map

  • Assigning different costs to semantic place labels lets the path planner avoid offices during working hours or choose the shortest route at night.The paper illustrates these alternatives with an A* path planner in a workplace with offices and corridors.Source: B. Path Planning on a Semantic Map · pp. 6–7

VI. THE PROVIDED ROS MODULE

The ROS module provides place categorization and semantic mapping without environment-specific training. The paper concludes that semantic maps support navigation behavior adapted to human needs and object recognition, and proposes extending the system to finer-grained categorization and visual place recognition.

VI. THE PROVIDED ROS MODULE

  • The ROS module combines a Places205 image-classification node with a mapping node that fuses semantic probabilities, laser data, pose, and a SLAM grid into a 3D semantic map.The package is described as deployable without environment-specific training; Places205 processing takes 0.031 seconds per 320 × 240 image on a Quadro K4000 GPU, with an additional 3 ms per one-vs-all classifier.Source: §VI · p. 7

VII. CONCLUSION AND FUTURE WORK

  • The conclusion presents semantic mapping as supporting human-compliant navigation and improved robotic object detection and recognition.The path-planning example assigns different costs to place labels so the robot avoids offices during working hours but takes the shortest path at night.Source: §VII · pp. 7–8, Figure 7 · p. 7
  • Future work will explore multi-scale or sub-scene labels, semantic-map guidance for visual place recognition, and additional real-world robotic tasks.The proposed place-recognition approach partitions the search space to semantically similar places.Source: §VII · p. 8
Loading 1507.02428v1…