PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Understanding for Field and Service Robots in a Priori Unknown Environments

Matthew R. Walter, Siddharth Patki, Andrea F. Daniele, Ethan Fahnestock, Felix Duvallet, Sachithra Hemachandra, Jean Oh, Anthony Stentz, Nicholas Roy, Thomas M. Howard

arXiv:2105.10396v2cs.ROcs.CL

TL;DR

Most language-grounding methods rely on prior spatial-semantic maps, limiting their scope in unknown or partially observed environments. This framework uses instruction-implied information to maintain a distribution over possible maps and infer behaviors; experiments report guided exploration and faster task-execution runtimes than opportunistic exploration.

  • Problem

    Most contemporary language-grounding approaches require detailed prior spatial-semantic maps, leaving instruction-following in unseen or partially observed environments insufficiently addressed.

    Source: §7 · p. 33

  • Method

    The framework treats language as a sensor, using implicit environmental information with robot sensing to maintain a world-model distribution and learn a belief-space policy for instruction-following.

    Source: §1 · pp. 2–3, §3.2 · p. 9

  • Results

    The framework enables instruction-following in unseen or partially observed environments and provides guided exploration with faster task-execution runtimes than opportunistic exploration.

    Source: §6 · p. 32

  • Takeaways & Limitations

    Spatial relations implicit in instructions can support more informed, deliberate paths to goals.

    Source: §5.1.2 · p. 25

    A reported failure occurs when the robot never confirms an inferred cone, allowing the incorrect hydrant to be labeled as the goal.

    Source: §5.1.1 · p. 22

Abstract

from arXiv · show

Contemporary approaches to perception, planning, estimation, and control have allowed robots to operate robustly as our remote surrogates in uncertain, unstructured environments. This progress now creates an opportunity for robots to operate not only in isolation, but also with and alongside humans in our complex environments. Realizing this opportunity requires an efficient and flexible medium through which humans can communicate with collaborative robots. Natural language provides one such medium, and through significant progress in statistical methods for natural-language understanding, robots are now able to interpret a diverse array of free-form commands. However, most contemporary approaches require a detailed, prior spatial-semantic map of the robot's environment that models the space of possible referents of an utterance. Consequently, these methods fail when robots are deployed in new, previously unknown, or partially-observed environments, particularly when mental models of the environment differ between the human operator and the robot. This paper provides a comprehensive description of a novel learning framework that allows field and service robots to interpret and correctly execute natural-language instructions in a priori unknown, unstructured environments. Integral to our approach is its use of language as a "sensor" -- inferring spatial, topological, and semantic information implicit in the utterance and then exploiting this information to learn a distribution over a latent environment model. We incorporate this distribution in a probabilistic, language grounding model and infer a distribution over a symbolic representation of the robot's action space. We use imitation learning to identify a belief-space policy that reasons over the environment and behavior distributions. We evaluate our framework through a variety navigation and mobile-manipulation experiments.

1 Introduction

Field and service robots need intuitive ways to work alongside people, but teleoperation burdens operators and most language-grounding methods depend on detailed prior maps. This work treats language as an environmental sensor and combines inferred environment information with robot sensing and imitation learning to support instruction-following in unknown environments.

1 Introduction

  • Teleoperation places significant cognitive load on operators, while spoken language offers an intuitive command medium without specialized interfaces or significant training.The paper motivates language as a way for robots to work with people between full autonomy and supervised teleoperation.Source: §1 · p. 2
  • Most contemporary language-grounding methods require a detailed prior world model, limiting their use when the robot’s environment is unknown or partially observed.Without that map, referents such as “the ball” and “the box” cannot be associated with specific locations.Source: §1 · p. 2
  • The framework treats language as a sensor, using spatial, topological, and semantic information implicit in instructions to support grounding in unknown environments.It learns environment observations and desired behaviors from commands.Source: §1 · pp. 2–3
  • An estimation-theoretic algorithm combines language-based observations with robot sensor streams to maintain a probability distribution over possible world models.An imitation-learning approach then learns a belief-space policy to choose navigation and manipulation actions given that distribution.Source: §1 · p. 3

2 Related Work

Prior-map language grounding has broadened from engineered correspondences to data-driven models, but many approaches still depend on spatial-semantic maps. This work differs from alternatives for unknown environments by reasoning over a probabilistic world-model distribution informed by the instruction.

2 Related Work

  • Early symbol-grounding systems used manually engineered correspondences and compact grammars, whereas statistical approaches learn broader linguistic, spatial, and semantic features from data.Many data-driven grounding methods still require a prior spatial-semantic map, often built with SLAM and manual annotation.Source: §2 · p. 3
  • Duvallet et al. (2013) build a deterministic map during navigation, while this approach uses imitation learning with a probabilistic environment model that makes instruction-derived information explicit.Neural navigation methods instead map language directly to actions without explicitly inferring a distribution over world models.Source: §2 · pp. 4–5
  • The framework assumes the concepts are known but their instantiations in the environment are unknown, unlike methods that learn novel linguistic predicates or ask users to resolve ambiguity.Source: §2 · p. 5
  • Unlike active-search approaches that often assume goal direction and distance are known, this framework maintains an explicit distribution over seen and unseen world states for grounding and planning under uncertainty.Prior linguistic mapping work uses free-form utterances that explicitly describe the environment; this approach infers information implicit in instructions and jointly considers mapping and planning.Source: §2 · p. 5

3 Technical Approach

The paper frames natural-language understanding as probabilistic symbol grounding: selecting referent symbols from a robot’s symbolic state and action spaces given an utterance and world model. This framing assumes a sufficiently complete world model, which excludes novel or partially observed environments.

3 Technical Approach

  • The symbol space includes environment concepts such as object locations and categories, as well as symbolic robot behaviors such as manipulating an object or navigating to a location.Source: §3 · p. 6
  • Grounding is framed as maximum a posteriori inference over sets of referent symbols, conditioned on the utterance parse and the robot’s world model.Source: §3 · p. 6
  • Assuming a complete prior world model precludes accurate grounding in novel or partially observed environments where that model is empty or incomplete.Source: §3 · p. 6

3.1 Approach Overview

The approach treats language understanding in unknown environments as coupled inference over environment annotations, a latent world-model distribution, and robot behaviors. A belief-space policy selects high-level actions as sensor observations and utterances update the model.

3.1 Approach Overview

  • A learned grounding model converts a parsed instruction into environment annotations that serve as observations about object and room existence, names, and relative locations.The framework assumes instructions are grammatically correct.Source: §3.1 · p. 6
  • Language-derived annotations and traditional sensor observations jointly inform a distribution over possible world models, including hypotheses about unseen objects and their locations.The world-model distribution also fuses sensor measurements and odometry with information implicit in utterances.Source: §3.1 · p. 6
  • The method infers behaviors conditioned on world models and the current utterance, then uses a learned belief-space policy to select a high-level action.A motion planner converts that action into control inputs, and the process repeats as new observations arrive until the instruction is satisfied.Source: §3.1 · pp. 6–7, Figure 3 · p. 7
  • To make inference tractable, the framework approximates map and behavior distributions with discrete samples and maintains the world-model distribution using Rao-Blackwellized particle filtering.Source: §3.1 · p. 7

3.2 Natural-Language Understanding

The framework treats language understanding in unknown environments as inference over both hypothesized world models and robot behaviors. Its language-grounding model uses parse-structured probabilistic inference to extract environment annotations and desired actions from instructions.

3.2 Natural-Language Understanding

  • A Distributed Correspondence Graph (DCG) uses parse structure and conditional probabilities to infer correspondences between language, symbols, and the environment model.It learns concept probabilities for linguistic structures such as nouns, prepositions, and verbs, assuming conditional independence across linguistic and symbolic constituents.Source: §3.2 · pp. 7–8
  • When a requested object or relation is absent from the robot’s model, language can supply the missing information and expose the mismatch with the user’s richer world model.The framework first uses utterance information to build a distribution over world models containing hypothesized objects and spatial relationships, then infers behaviors using that distribution and sensor observations.Source: §3.2 · pp. 8–9
  • Annotation inference represents object classes, locations, spatial-relation classes, regions, and relations independently of a particular environment model.These symbols allow language-derived relationships, such as an object being inside another, to inform the distribution over objects and locations in a partially observed environment.Source: §3.2 · p. 9
  • Behavior inference uses grounded objects, regions, spatial relations, modes, and actions to represent candidate robot behaviors.Modes such as “safe” and “quickly” can produce different actions for the same goals by influencing plan costs.Source: §3.2 · pp. 9–10
  • The annotation symbol space ΓA is the union of object, spatial-relation, region, and relation symbols.The experimental symbolic representations can be adjusted to the diversity of tasks, while grounding language against inferred object sets is not studied here.Source: §3.2 · p. 10

3.3 Semantic Mapping

Semantic mapping combines language-derived annotations with robot observations to maintain a distribution over possible metric, topological, and semantic world models. A particle filter samples candidate topologies, updates their pose distributions, and weights them against observations.

3.3 Semantic Mapping

  • The semantic graph combines topology, a metric pose map, and semantic region categories, with nodes for observed or language-hypothesized objects and places.Nodes are grouped into spatially coherent regions; edges encode robot motion, observations, or language-inferred spatial relations.Source: §3.3 · p. 10, Figure 5 · p. 11
  • Language annotations supply hypothesized entity existence, type, and relative location, and are treated alongside sensor data as observations for estimating world models.The framework maintains a distribution over semantic maps using a Rao-Blackwellized particle filter, sampling topologies and representing the metric pose distribution as a Gaussian.Source: §3.3 · pp. 10, 12
  • The filter assumes only a few topologies have meaningful likelihood; in open areas with sparsely distributed objects, tracking more topologies raises computational cost.Perceptual aliasing can also undermine the assumption that most candidate topologies will be inconsistent with observations.Source: §3.3 · pp. 11–12

3.3.1 The Proposal Distribution

  • The proposal distribution modifies each particle’s graph using robot motion, sensor observations, and language annotations, including region segmentation and hypothesized spatial relations.Spatially distinct areas are more likely to be represented as separate regions, while connected regions with the same modal category may be merged.Source: §3.3.1 · pp. 13–14

3.3.2 Updating the metric map based on new edges

  • New graph edges update the Gaussian pose distribution, whose information-matrix sparsity follows the topology and supports more efficient inference.The method incorporates relative-pose constraints from new edges, including odometry edges between successive poses.Source: §3.3.2 · p. 15

3.3.3 Re-weighting particles and resampling

  • Particles are reweighted by the likelihood of language annotations and region-appearance observations, then resampled when weight variance exceeds a threshold.Appearance likelihoods penalize particles whose region types conflict with known traversed regions; the experiments set the effective-particle threshold to half the particle count.Source: §3.3.3 · p. 16

3.4 Planning Under Uncertainty

Planning treats instruction-following in an unknown environment as decision-making under uncertainty over semantic maps. The policy represents action-feature distributions and is trained from expert demonstrations to reason over those distributions.

3.4 Planning Under Uncertainty

  • The planner minimizes one-step expected cost over belief-space actions, including paths to graph landmarks paired with their distributions and an explicit stop action.This representation lets it explore hypothesized locations, backtrack after unfruitful actions, or stop when it judges the instruction satisfied.Source: §3.4 · p. 16

3.4.1 Belief-Space Reasoning using Distribution Embedding

  • Kernel mean embeddings represent distributions over action features through their first K moments, allowing the policy to compare feature distributions across possible maps.Features include path shape, landmark geometry, and action-landmark relationships; features are computed for each hypothesized landmark sample and aggregated into moments.Source: §3.4.1 · p. 17, Figure 6 · p. 17

3.4.2 Imitation Learning Formulation

  • The policy is learned by minimizing a weighted sum of feature moments, using subgradient updates and DAgger’s iterative collection of policy states and expert corrections.When the policy disagrees with the demonstration, the update shifts feature weights between the demonstrated and planned actions.Source: §3.4.1 · p. 18, §3.4.2 · p. 19
  • Imitation learning trains a belief-space policy to predict an expert’s action among the available actions without assuming prior knowledge of the environment.The formulation treats action prediction as multiclass classification and penalizes disagreement with the expert using a margin-based surrogate loss.Source: §3.4.2 · p. 18

4 Experiments

The experiments evaluated natural-language instruction following on three physical robotic systems. The systems used distinct symbolic representations and language-grounding corpora for navigation and mobile manipulation.

4 Experiments

  • The experiments used three physical platforms: a Husky UGV, a robotic wheelchair, and a Husky fitted with a UR5 manipulator.The UGV and wheelchair were used for route instructions, while the mobile manipulator was used for mobile-manipulation instructions.Source: §4 · p. 19, Figure 7 · p. 19
  • The UGV experiments used seven spatial relations, eight object types, two behavior actions, three modes, and a DCG model trained on 39 labeled examples.The representation included no location types.Source: §4 · p. 19
  • The wheelchair experiments used 12 spatial relations, eight object types, nine location types, four behavior actions, three modes, and an HDCG model trained on 54 labeled examples.They were conducted in an office-building environment.Source: §4 · pp. 19–20
  • The mobile manipulator was tested indoors and outdoors on instructions requiring navigation and object manipulation, using a DCG model trained on 115 labeled examples.Its symbolic representation included object, location, and spatial-relation types for annotation and behavior inference.Source: §4 · p. 20

5 Experimental Results

The experiments evaluate navigation in simulation and on physical robots, then test mobile manipulation in unknown environments. Results show how language-informed environment estimates support instruction following, while detailed perception and reasoning can be time-consuming.

5.1.1 Monte Carlo Simulations

  • Using spatial relations in commands improves success and path efficiency over commands without explicit relations; performance approaches the Known Map system as field-of-view increases.Figure 9 shows success rising and distance traveled falling with greater sensing range.Source: §5.1.1 · pp. 21–22, Figure 9 · p. 22
  • A failure occurs when the robot sees a hydrant first and hypothesizes an unconfirmed cone, which can make the wrong hydrant the goal.The described case involves “nearest to the cone” in an environment with two hydrants.Source: §5.1.1 · p. 22
  • Our method reaches multi-room navigation goals along paths only slightly longer than the Known Map baseline, but takes significantly more time than that baseline.It outperforms the Without Language baseline in path length and execution time; intermediate-goal stops and implementation pauses contribute to the added time.Source: Table 2 · p. 23, §5.1.1 · p. 23

5.1.2 Physical Experiments

  • 8 of 10 experiments identified the correct hydrant for “behind the cone” with two hydrants; “nearest” succeeded 10/10 at 4 m but only half at 3 m.The Husky’s average distance was 8.1 m (σ = 1.3 m) with the method and 8.4 m (σ = 0.6 m) with the Known Map baseline.Source: Table 3 · p. 24, §5.1.2 · p. 25
  • In the wheelchair kitchen task, all three methods reached the goal; the proposed method first favored the wrong hallway, then updated its map and reached the correct one.The method was tested in six trials; the Known Map baseline had three trials and the Without Language baseline had five.Source: §5.1.2 · pp. 26–27, Table 4 · p. 27

5.2 Instruction Following for Mobile Manipulation

  • Approximately six minutes indoors and ten outdoors; perception consumed a large fraction of task time, and richer world models increased perception and symbol-grounding costs.The runtime analysis covered three indoor and three outdoor trials.Source: §5.2 · pp. 29, 32, Table 5 · p. 32

6 Discussion

The framework supports instruction-following in previously unseen or partially observed environments and faster task execution than opportunistic exploration. Its scope and scalability remain constrained by small evaluation environments, costly detailed world models, and fully annotated training data.

6 Discussion

  • Experiments in simulation and on robots show instruction-following in unseen or partially observed environments, with faster task execution than opportunistic exploration.The evaluated environments are small relative to typical field and service robotics settings; larger-scale evaluation remains future work.Source: §6 · p. 32
  • Reasoning over highly detailed world-model distributions is computationally expensive for both perception and symbol grounding.Adaptive-perception work cited here reports approximately 50% lower task-execution runtimes in indoor and outdoor experiments.Source: §6 · p. 32
  • The approach relies on fully annotated data, and densely labeling examples for annotation and behavior inference is nontrivial.The cup-on-table example requires labeling objects, their relation, and the semantic correspondence between physical relationships and labels.Source: §6 · pp. 32–33
  • The inference procedures do not bound the worlds or language in which their expressed symbols remain valid, and speech, parsing, or perception noise may affect inference.The discussion identifies confidence evaluation as relevant to deciding whether to use annotations or confirm them through dialogue.Source: §6 · p. 33

7 Conclusions

The framework supports natural-language instruction-following in previously unseen or partially observed environments, including navigation and mobile manipulation. Experiments report faster task execution than a language-grounding baseline that does not use instruction-provided environment information, while exposing computational and scale limitations.

7 Conclusions

  • Experiments indicate faster task execution in previously unseen environments than a language-grounding baseline that does not use instruction-provided environment information.The reported experiments include simulation and three robotic platforms.Source: §7 · p. 33
  • The method supports tasks involving both navigation and mobile manipulation in previously unseen or partially observed environments.Source: §7 · p. 33
  • Maintaining and reasoning over distributions of highly detailed world models is computationally expensive for perception and symbol grounding.The conclusion frames efficient representation of complex environments while supporting diverse tasks at large scale as an open research question.Source: §7 · p. 33
Loading 2105.10396v2…