Source-linked AI summary
A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution
Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, Yoav Artzi
TL;DR
High-level natural-language instructions require robots to reason across abstraction layers and long horizons to generate low-level actions. The paper introduces a persistent spatial semantic representation with hierarchical control, achieving state-of-the-art ALFRED performance without low-level instructions.
Problem
Mapping high-level natural-language instructions to low-level mobile-manipulation actions requires long-horizon reasoning across abstraction layers.
Method
HLSM continually updates a spatial semantic world representation and uses hierarchical controllers to generate subgoals and action sequences.
Results
10.04% absolute improvement in SR on ALFRED’s test unseen split over HiTUT G-only, with state-of-the-art performance across seen and unseen environments using only high-level instructions.
Takeaways & Limitations
Persistent spatial semantic representations support hierarchical execution of long-horizon mobile-manipulation tasks from high-level instructions alone.
Takeaways & Limitations
Physical deployment would require changes to the low-level controller and evaluation of robustness to errors in continuous environments.
Abstract
from arXiv · showhide
Natural language provides an accessible and expressive interface to specify long-term tasks for robotic agents. However, non-experts are likely to specify such tasks with high-level instructions, which abstract over specific robot actions through several layers of abstraction. We propose that key to bridging this gap between language and robot actions over long execution horizons are persistent representations. We propose a persistent spatial semantic representation method, and show how it enables building an agent that performs hierarchical reasoning to effectively execute long-term tasks. We evaluate our approach on the ALFRED benchmark and achieve state-of-the-art results, despite completely avoiding the commonly used step-by-step instructions.
1 Introduction
The paper addresses mapping high-level natural-language tasks to low-level mobile-manipulation actions, proposing a persistent spatial semantic representation and hierarchical reasoning to support long-horizon execution. HLSM achieves state-of-the-art ALFRED performance without using low-level instructions during training or testing.
- 1 Introduction: High-level instructions improve communication efficiency but require long-horizon reasoning to map abstract language onto low-level actions.The task involves exploration, locating relevant objects, distributing actions across multiple objects, and mapping verbs to the robot’s action space.
- 1 Introduction: HLSM uses a continually updated spatial semantic world representation as long-term memory for navigation and reasoning about object relations.The representation retains environmental spatial properties and supports hierarchical language-conditioned mobile manipulation.
- 1 Introduction: The approach achieves state-of-the-art performance on ALFRED, outperforming methods that use detailed sequential instructions.Low-level instructions are excluded during both training and testing.
- 1 Introduction: The work contributes a modular representation-learning approach for mapping high-level natural-language task descriptions to actions in interactive 3D environments.The broader setting is mobile manipulation in home environments, where robots must explore and infer action sequences over long horizons.
2 Related Work
Prior robotics work largely used simple synthetic goals or fully specified step-by-step instructions, while symbolic and learned representations each present tradeoffs. This work instead integrates a 3D semantic voxel map into hierarchical representation learning for high-level instruction following and reports state-of-the-art benchmark results.
- Related instruction-following work: Prior instruction-following research mostly addressed simple synthetic goals or natural-language instructions that explicitly detailed every intermediate step, unlike this focus on high-level instructions.The paper motivates high-level instructions as more likely in home environments.
- Robot representations: Symbolic representations improve grounding and robustness but rely on ontologies that are laborious to scale, motivating learned alternatives for world state, action history, and language semantics.
- Proposed representation: The proposed system integrates a semantic voxel-map state representation into hierarchical representation learning, extending prior semantic 2D maps to 3D for mobile-manipulation instruction following.The design relates to sparse metric, topological, and semantic maps used in robotics.
- Proposed representation: The extended 3D map achieves state-of-the-art results on a challenging mobile-manipulation benchmark.
3 Problem Definition
The problem is to generate an action-state execution from a natural-language instruction and initial world state under environment dynamics. Success requires satisfying all task goal conditions, with partial success measured by the achieved fraction.
- The agent generates an execution of states and actions from instruction L and initial state s0, with each next state determined by environment dynamics T.The state captures the environment layout and object poses and states.
- Observations combine a first-person RGB image, the agent’s pose, a one-hot encoding of the held object class, and the instruction.
- Task success requires all goal conditions corresponding to instruction L to hold in the final state, while partial success is their achieved fraction.
- ALFRED uses seen and unseen environments, with parameter-free navigation actions and mask-parameterized interaction actions.Navigation includes MOVEAHEAD, ROTATELEFT, and ROTATERIGHT; interactions include PICKUP, PUT, TOGGLEON, TOGGLEOFF, OPEN, CLOSE, and SLICE.
- The agent’s pose and held-object encoding are computed by dead-reckoning from RGB observations and actions.
4 Hierarchical Model with a Persistent Spatial Semantic Representation
The model hierarchically executes language instructions through high- and low-level controllers that share a persistent spatial state representation. This representation accumulates semantic observations and supports long-horizon planning, near-term reasoning, and object-oriented interaction.
- The policy combines an observation model with high- and low-level controllers that use a shared cumulative spatial state representation for planning and near-term reasoning.The high-level controller plans long-horizon subgoals, while the low-level controller executes each subgoal until success or failure.
- The high-level controller predicts interaction subgoals defined by an interaction type, argument class, and 3D mask identifying the target instance.It conditions subgoal prediction on the instruction, past subgoals, and current spatial state.
- The persistent state encodes observed-object locations, visibility, the held object class, and the agent pose in a semantic 3D voxel map.It includes a semantic map, observability map, inventory vector, and pose.
- Seven top-down affordance classes provide object-class-agnostic spatial features for reasoning about actions such as picking up an object.The affordances are pickable, receptacle, togglable, openable, ground, obstacle, and observed.
- The observation model updates this state through perception, 3D projection, and accumulation, retaining information from voxels that are not currently visible.It predicts semantic segmentation and depth, projects observations into voxels, and integrates them into the persistent representation.
5 Learning
The policy learns four models independently from supervised demonstrations: perception, high-level subgoal prediction, and low-level navigation. Training uses derived perception, subgoal, and navigation datasets with task-specific loss functions.
- Learning: The policy independently trains four learned models—segmentation, depth, πH, and NAVMODEL—using supervised learning on demonstrations from seen environments.The demonstrations pair high-level natural-language instructions with execution sequences of states and actions.
- Learning: Training processes the demonstrations into perception, subgoal, and navigation datasets containing visual labels, language-conditioned subgoal context, and navigation states with poses.State representations for the subgoal and navigation datasets are constructed with the observation model using ground-truth depth and segmentation.
- Learning: Perception models and πH optimize cross-entropy losses, while NAVMODEL uses cross-entropy for positions and yaw and an L2 loss for pitch.πH predicts the k-th subgoal from the subgoal dataset, and NAVMODEL is trained on navigation states, subgoals, and poses.
6 Experimental Setup
The experiments evaluate HLSM on ALFRED using high-level instructions, compare it with leaderboard systems and step-by-step-instruction approaches, and analyze perception, high-level control, and exploration through ablations and sensory oracles.
- Environment, Data, and Evaluation: The evaluation uses ALFRED, where agents map raw RGB observations to actions for tasks specified by high-level natural language instructions.ALFRED includes 108 training scenes, 88/4 validation seen/unseen scenes, and 107/8 test seen/unseen scenes, with 21,023 training tasks and separate validation and test task counts.
- Systems: HLSM is compared with high-level-instruction systems, especially HiTUT, and with approaches that use step-by-step instructions.HiTUT models hierarchical task structure with flat BERT without a spatial representation.
- Ablations and Sensory Oracles: The study evaluates perception through sensory oracles providing ground-truth depth, segmentation, or both.These comparisons are intended to analyze the observation model.
- Ablations and Sensory Oracles: High-level controller ablations remove the subgoal encoder, language encoder, or representation encoder while retaining selected state-representation predictions.The representation encoder is used to predict subgoal type and argument class, while the state representation predicts the subgoal argument mask.
- Ablations and Sensory Oracles: A low-level controller ablation removes the exploration procedure.
7 Results
The approach achieves state-of-the-art performance on ALFRED using only high-level instructions, including substantial success-rate gains over high-level and step-by-step baselines. Development analyses show the roles of perception, spatial reasoning, exploration, and several failure modes.
- Test results: 4.84% absolute (31.4% relative) higher SR than ABP on test unseen shows competitiveness with methods using low-level instructions, while ABP and LWIT perform better on test seen.The seen-split difference is attributed to potentially stronger scene overfitting.
- Development results: Ground-truth segmentation improves validation SR by 6.6% in seen and 16.4% in unseen scenes, while adding both segmentation and depth yields 11.1% and 21.9% gains.Ground-truth depth alone does not significantly affect performance; five full-model runs give a 1.1% absolute validation-unseen SR standard deviation.
- Ablations: Removing the language encoder performs poorly, whereas subgoal-history inputs and random exploration can be removed without significant performance effects, highlighting the importance of spatial information for grounding.Predicting subgoal types and argument classes can sometimes proceed without spatial reasoning, but grounding subgoals requires spatial information.
- Test results: 10.04% absolute (98.1% relative) and 11.53% absolute (62.6% relative) SR improvements over HiTUT G-only yield state-of-the-art performance on test unseen and seen splits, respectively.These results use only high-level instructions and are reported in Table 1.
- Failure analysis: Figure 4 identifies failures from perception, insufficient exploration, navigation, subgoal prediction, and limited state-aware multi-step planning.Examples include missing or extraneous obstacles, wrong-object pickup, failure to search cabinets, blocked openings, and subgoal prediction errors.
8 Discussion and Limitations
The persistent spatial semantic representation enables hierarchical instruction-following performance at the state of the art, while remaining limited by exploration, perception generalization, and low-level motion planning.
- The persistent spatial semantic representation enables a hierarchical model to achieve state-of-the-art performance on a challenging instruction-following mobile manipulation task.
- Key bottlenecks are long-horizon exploration, perception generalization to unseen environments, and low-level motion planning for continuous collision avoidance.
- Joint reinforcement-learning training of πH, πL, and observation model F could improve performance.
A Appendix · A.1 Frequently Asked Questions
The FAQ clarifies that step-by-step instructions are unnecessary, while outlining real-robot deployment requirements, scalability limits, modeling assumptions, localization considerations, evaluation choices, and subgoal-sampling benefits.
- A.1 Frequently Asked Questions: Sequential step-by-step instructions are unnecessary during both training and test-time.
- A.1 Frequently Asked Questions: Applying the approach to a real robot requires continuous-motion low-level control in cluttered environments and an ALFRED manipulation interface.
- A.1 Frequently Asked Questions: The work motivates real-robot operation but does not claim high-level natural-language mobile manipulation from raw vision in unseen real environments.
- A.1 Frequently Asked Questions: Scaling to physically larger environments is mainly limited by semantic-memory capacity, with commodity hardware likely restricting deployment to interior scenes.
- A.1 Frequently Asked Questions: The low-level GOTO controller uses a value-iteration network with deterministic grid-navigation dynamics, while other environment dynamics remain unknown and unmodeled.
- A.1 Frequently Asked Questions: The representation assumes reliable pose estimates; distant voxels may tolerate errors better, whereas nearby voxels require precision for object-instance mask generation.
- A.1 Frequently Asked Questions: The full HLSM model was evaluated on the test set, despite the state-encoding ablation performing better in unseen validation environments because segmentation errors corrupt its encodings.
- A.1 Frequently Asked Questions: Subgoal sampling addresses both systematic failures from incorrect selections and random failures from unsuccessful low-level interaction poses.
A.2 Extended Related Work
Prior work bridges high-level language to long action sequences through temporal abstraction, symbolic grounding, or direct language-to-action models. This approach uses hierarchical controllers and a persistent spatial-semantic representation, extending dense semantic mapping to 3D voxel maps with explicit occupancy and observability tracking.
- Grounding High-level Language to Actions in Robotics: Prior symbolic methods align language constituents, world-model percepts, and subgoals, but require instructions to mention every subgoal rather than infer omitted intermediate goals.Other approaches instead map language directly to reward specifications or postconditions.
- Grounding High-level Language to Actions in Robotics: The approach predicts subgoals from high-level instructions and generates actions for each subgoal using hierarchical controllers grounded in a spatial-semantic state representation.The representation supports reasoning about which subgoals advance the task and which actions fulfill them.
- Semantic Maps for Language Grounding in Robotics: The method extends dense grid-based semantic mapping from 2D to 3D voxel maps while explicitly tracking occupancy and observability.This work builds on learned neural-network approaches for language grounding and earlier grid-based results on part of ALFRED.
- Semantic Maps for Language Grounding in Robotics: Dense spatial-semantic representations can be built in real time from RGBD data, capture complex indoor structures, and encode object relationships without a spatial ontology or instance tracking.The passage identifies a main limitation of the approach but does not provide its continuation.
- Detailed comparison to HiTUT: Compared with HiTUT, the approach uses hierarchical high- and low-level controllers with a shared spatial state representation, whereas HiTUT uses a flat transformer for joint subgoal planning and action prediction.Both approaches sample subgoals dynamically and backtrack after subgoal failure; their perception systems use segmentation and object detection models, respectively.
A.3 Observation Model Details … A.10 Additional Results
The appendix details perception, hierarchical execution, low-level control, training procedures, experimental settings, and additional qualitative outcomes. It specifies how visual observations become persistent spatial inputs and how subgoals are executed through navigation and interaction.
- A.3 Observation Model Details: The system predicts semantic segmentation and discretized depth, filters uncertain points with subgoal-aware confidence masks, and projects reliable observations into the voxel map.Depth uses 50 bins at 0.1m resolution, with 0.25m voxels; the depth resolution is kept below half the voxel size.
- A.4 Model Execution Flow: At each timestep, the execution loop updates the state, samples a high-level subgoal when needed, invokes the low-level policy, and advances or discards the subgoal after success or failure.The loop ends when the high-level policy emits gSTOP.
- A.5 High-Level Controller Details: REFINER combines egocentric spatial features, semantic voxel masks, agent pose, and instruction embeddings to predict a 3D mask for the selected subgoal argument.Its language-conditioned image-to-image backbone uses LingUNet and transforms spatial maps into the global reference frame.
- A.6 Low-Level Controller Details: The low-level controller explores until the target class is observed, samples an interaction pose, projects the predicted object mask into the camera view, and executes the interaction.Navigation uses a predefined Value Iteration Network over a 2D grid, while interaction success returns aPASS or aFAIL to the high-level controller.
- A.6.2 SampleExplorationPosition; A.6.3 SampleInteractionPose; A.6.4 InteractionMask: Exploration samples frontier positions from unoccupied floor or rug regions, while pose prediction samples position and yaw distributions and estimates camera pitch conditioned on the subgoal and state.The interaction mask combines class-based image segmentation with projected voxel evidence to identify the correct object instance.
- A.7 Additional Learning Details; A.7.1 Observation Model Learning: The observation models train on depth distributions and one-hot segmentation labels, using segmentation-aware augmentation to vary likely appearance classes and improve diversity across correlated scenes.Augmentation is applied with 50% probability, alongside horizontal flipping with 50% probability.
- A.8 Additional Experimental Details; A.9 Hyperparameters: Training demonstrations are modified to provide more informative first-person observations, and hyperparameters are hand-tuned on the validation unseen split.The supplied appendix passage describes inserted initial rotations but truncates the remaining trajectory modifications.
- A.10 Additional Results: Additional results consist of linked qualitative examples, including a failed execution example, while Table 5 reports the selected hyperparameter values.The appendix also provides the AUGMENT implementation used to modify segmented image regions.