Source-linked AI summary
GOAT: GO to Any Thing
Matthew Chang, Theophile Gervet, Mukul Khanna, Sriram Yenamandra, Dhruv Shah, So Yeon Min, Kavit Shah, Chris Paxton, Saurabh Gupta, Dhruv Batra, Roozbeh Mottaghi, Jitendra Malik, Devendra Singh Chaplot
TL;DR
Mobile robots need to navigate indefinitely toward goals expressed naturally across homes and warehouses, but prior systems often address only narrow modalities or settings. GOAT combines multimodal navigation with lifelong instance-aware memory in a modular, platform-agnostic system, achieving 83% overall success and improving with experience.
Problem
Prior navigation methods commonly specialize in one goal modality, one goal per episode, or simulation, limiting evaluation of lifelong multimodal navigation in the real world.
Method
GOAT uses a modular navigation system with an instance-aware semantic memory that stores object appearances, locations, and category semantics for image, language, and category goals.
Results
83% overall success across 90+ hours in 9 homes and 675 goals surpassed previous methods and ablations by 32% absolute, while performance rose from 60% initially to 90% after exploration.
Takeaways & Limitations
GOAT serves as a general navigation primitive that can be deployed across robot embodiments and applied to downstream tasks such as pick and place and social navigation.
Takeaways & Limitations
Open-vocabulary vision-and-language models performed significantly worse than a 2017 Mask R-CNN model in the real-world detection setting.
Abstract
from arXiv · showhide
In deployment scenarios such as homes and warehouses, mobile robots are expected to autonomously navigate for extended periods, seamlessly executing tasks articulated in terms that are intuitively understandable by human operators. We present GO To Any Thing (GOAT), a universal navigation system capable of tackling these requirements with three key features: a) Multimodal: it can tackle goals specified via category labels, target images, and language descriptions, b) Lifelong: it benefits from its past experience in the same environment, and c) Platform Agnostic: it can be quickly deployed on robots with different embodiments. GOAT is made possible through a modular system design and a continually augmented instance-aware semantic memory that keeps track of the appearance of objects from different viewpoints in addition to category-level semantics. This enables GOAT to distinguish between different instances of the same category to enable navigation to targets specified by images and language descriptions. In experimental comparisons spanning over 90 hours in 9 different homes consisting of 675 goals selected across 200+ different object instances, we find GOAT achieves an overall success rate of 83%, surpassing previous methods and ablations by 32% (absolute improvement). GOAT improves with experience in the environment, from a 60% success rate at the first goal to a 90% success after exploration. In addition, we demonstrate that GOAT can readily be applied to downstream tasks such as pick and place and social navigation.
1 Introduction
GOAT addresses real-world navigation by combining multimodal goals, lifelong object memory, and deployment across robot platforms. Its instance-aware semantic memory supports image and language queries, while experiments show strong performance that improves with environmental experience.
- 1 Introduction: The system design is motivated by cognitive maps, route-based learning, and the role of visual landmarks in animal and human navigation.These observations motivate spatial memory, continual improvement, and rich multimodal environmental representations.
- 1 Introduction: GOAT supports category labels, target images, and language descriptions while learning from past experience and deploying on different robot embodiments.The authors deploy it on both quadruped and wheeled robots.
- 1 Introduction: Its instance-aware semantic memory stores object appearances from multiple viewpoints alongside category semantics, distinguishing different instances of the same category.The memory is continually augmented as the robot spends more time in an environment.
- 1 Introduction: 83% overall success across 90+ hours, 9 homes, 675 goals, and 200+ object instances surpassed previous methods and ablations by 32% absolute.The evaluation covers category, image, and language navigation goals in real homes.
- 1 Introduction: 60% success at the first goal increased to 90% after exploration, showing improved navigation with environmental experience.The system remembers object locations and can query whether a goal object has already been encountered.
- 1 Introduction: GOAT targets a gap in prior navigation systems that commonly specialize in one modality, one goal, simulation, or rudimentary real-world settings.The paper evaluates multiple goal specifications sequentially in real-world environments.
2 Results
GOAT is evaluated as a multimodal, lifelong navigation system in unseen real homes, using instance-aware memory to match goals and improve through experience. Across navigation benchmarks and downstream applications, it supports efficient object localization, rearrangement, and social navigation on multiple robot platforms.
- GOAT Agent: GOAT matches language goals with CLIP image-language similarity and image goals with SuperGLUE keypoint matching over views stored in Object Instance Memory.The system maintains a top-down semantic map and stores multiple observed images for individual object instances.
- Navigation Performance in Unseen Natural Home Environments: 83% average success across 90 episodes in 9 unseen homes, with category, image, and language goals evaluated against three baselines.The benchmark covered 5–10 object instances per episode and 200+ distinct instances; Table 1 reports success rate and SPL per goal instance.
- Navigation Performance in Unseen Natural Home Environments: GOAT achieves 86% success on image goals and 68% on language goals, while CLIP on Wheels reaches 51% overall success and GOAT without Instances reaches 49%.The results identify language-goal localization as harder than image-goal localization and quantify the benefit of instance-aware matching.
- Navigation Performance in Unseen Natural Home Environments: 90% success (0.80 SPL) for goals 5–10 after exploration, versus 60% success (0.20 SPL) at the first goal, showing improvement with environmental experience.GOAT without memory shows no improvement with experience, while COW improves but plateaus at lower performance.
- Navigation Performance in Unseen Natural Home Environments: GOAT localizes all five goals with 0.78 SPL in a representative trajectory, compared with one of five for CLIP on Wheels and 0.40 SPL without memory.Without memory, the agent re-explores the environment for every goal.
- Applications: GOAT extends to rearrangement and social navigation, finding objects and receptacles at 79% and 87% success and following people at 83% success.For social navigation, it updates the map as people move and preserves 81% success when treating moving people as dynamic obstacles.
3 Discussion
GOAT combines modular components with instance-aware memory to support robust, general-purpose navigation across goal types and robot platforms. Discussion findings identify experience-dependent matching limits, stronger image than language matching, category filtering and context benefits, and open-vocabulary detection challenges.
- Modularity allows GOAT to Achieve Robust General-Purpose Navigation in the Real World: 83% success across image, language, and category goals demonstrates GOAT’s robust general-purpose navigation, reaching 90% after full exploration.The modular system combines learned components with classical mapping and planning through object-instance memory.
- Modularity allows GOAT to Achieve Robust General-Purpose Navigation in the Real World: GOAT can transfer across robot embodiments without new data collection or retraining when platforms provide RGB-D sensing, pose, and basic locomotion.This modular deployment contrasts with end-to-end methods that require retraining for each embodiment.
- Matching Performance During Exploration Lags Behind Performance After Exploration: During exploration, fixed matching thresholds can miss true positives and admit false positives, producing performance 6% below post-exploration matching.After exploration, the agent can select the instance with the best matching score instead of relying on the threshold.
- Image Goal Matching is More Reliable than Language Goal Matching: Image-goal matching is 23% more successful than language-feature matching because geometric correspondences capture fine-grained instance properties that CLIP may miss.The comparison is between SuperGLUE-based image matching and semantic features from language descriptions.
- Goal Matching Improves by Subsampling Instances by Category and Adding Context: Filtering comparisons to the goal category improves matching accuracy by 23%, while wider bounding boxes improve accuracy by up to 22% over object boxes alone.Category filtering also reduces computation, and added background context helps with both SuperGLUE and CLIP matching.
- Real-World Open-Vocabulary Detection: Limitations and Opportunities: Open-vocabulary vision-language models perform significantly worse than a 2017 Mask R-CNN in real-world detection, highlighting an embodied robustness gap.The paper hypothesizes trade-offs between versatility and robustness, plus limited representation of embodied interaction data in web-scale training.
4 Materials and Methods
GOAT combines RGB-D perception, semantic mapping, instance-aware memory, and hierarchical policies to navigate toward multimodal goals while exploring when targets are not yet localized. Its modular design supports deployment across different robot platforms and continual accumulation of object-instance views.
- 4.1 Go To Any Thing System Architecture: GOAT detects object instances from RGB-D input, projects them into a top-down semantic map, and stores their views in Object Instance Memory.Depth holes are filled using monocular depth estimates grounded by sensed depth before constructing the semantic map.
- 4.1 Go To Any Thing System Architecture: The semantic map records object instances, obstacles, explored areas, and the agent’s current and past locations in a spatial grid.The map uses K = C + 4 channels, with the first C channels storing unique instance identifiers.
- 4.1 Go To Any Thing System Architecture: Object Instance Memory clusters detections by semantic category and overlapping map locations, updating matched instances or adding new ones over time.Each instance retains map cells, contextual bounding-box views, and its semantic category, enabling accumulated views of specific objects.
- 4.1 Go To Any Thing System Architecture: The global policy matches category, language, or image goals against memory and issues an exploration goal when no target instance is localized.Category goals use semantic-map presence, while language and image goals use modality-specific matching; localized targets become long-term map navigation goals.
- 4.1 Go To Any Thing System Architecture: The local policy plans paths to global goals with the Fast Marching Method and executes them using platform-specific or deterministic waypoint controllers.GOAT was deployed on Boston Dynamics Spot and Hello Robot Stretch, with large-scale quantitative evaluation conducted on Spot in nine homes.
- 4.1 Go To Any Thing System Architecture: Evaluation covered 90 hours in nine unseen homes, using 10 trajectories per home and goals from 15 object categories.Success required stopping within 1 meter of the correct goal instance within 200 robot steps; SPL was also computed per goal.
5 Supplementary
The supplementary analysis evaluates matching choices for language and image goals using manually annotated trajectories and reports which strategies most reliably identify target instances. It also provides qualitative examples of goal-instance matching and parameter ablations.
- Matching Strategy Analysis: SuperGLUE-based image keypoint matching is 13% more successful on average than CLIP feature matching for image goals.The comparison helps explain GOAT’s advantage over COW, which uses CLIP feature matching.
- Matching Strategy Analysis: Adding a nonzero matching threshold makes performance 6% worse on average than using no threshold.The threshold is used during exploration to decide whether to accept a current match or continue searching, while zero threshold is used after full exploration.
- Matching Strategy Analysis: Category-based instance sub-sampling is 23% more successful on average than comparing against all stored instances.The strategy improves precision and efficiency when object detection supplies the correct category.
- Matching Strategy Analysis: Matching padded instance crops is 4.6% better than full-image matching and 22% better than object bounding boxes across settings.The best setting includes surrounding context without using the entire scene image.
- Matching Strategy Analysis: Maximum matching score selection is 2% to 16% better than median, average, and top-2 average aggregation across settings.The analysis compares alternative ways to aggregate multiple views of multiple instances.
- Matching Strategy Analysis: Image-to-image matching is 23% better on average than language-to-image matching across settings.The supplementary discussion attributes this difference to stronger geometric correspondences for image matching than fine-grained semantic representations from CLIP.