Source-linked AI summary
GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation
Mukul Khanna, Ram Ramrakhya, Gunjan Chhablani, Sriram Yenamandra, Theophile Gervet, Matthew Chang, Zsolt Kira, Devendra Singh Chaplot, Dhruv Batra, Roozbeh Mottaghi
TL;DR
Navigation research has often focused on single-modality goals and episodic settings, although users may need both varied goal specifications and agents that use past experience. GOAT-Bench evaluates open-vocabulary sequences of category, image, and language goals and compares modular and end-to-end methods. SenseAct-NN methods have higher success but worse path efficiency than modular methods, and memory improves efficiency for both classes in later subtasks.
Problem
Many navigation methods handle only one goal modality, and prior work often studies episodic settings rather than agents using past experience in the same environment.
Source: §1 · p. 2Method
GOAT-Bench is a reproducible benchmark for open-vocabulary lifelong navigation to sequences of category, image, or language goals, comparing modular and end-to-end methods with and without memory.
Source: §1 · p. 2Results
SenseAct-NN methods achieve 2.9–4.6% higher success rates but 4.7–9.2% worse SPL than modular methods; memory improves later-subtask efficiency for both classes.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
The Embodied AI community has made significant strides in visual navigation tasks, exploring targets from 3D coordinates, objects, language descriptions, and images. However, these navigation models often handle only a single input modality as the target. With the progress achieved so far, it is time to move towards universal navigation models capable of handling various goal types, enabling more effective user interaction with robots. To facilitate this goal, we propose GOAT-Bench, a benchmark for the universal navigation task referred to as GO to AnyThing (GOAT). In this task, the agent is directed to navigate to a sequence of targets specified by the category name, language description, or image in an open-vocabulary fashion. We benchmark monolithic RL and modular methods on the GOAT task, analyzing their performance across modalities, the role of explicit and implicit scene memories, their robustness to noise in goal specifications, and the impact of memory in lifelong scenarios.
1. Introduction
Navigation benchmarks and methods have largely targeted individual goal modalities and episodic tasks, despite the need for varied goal specifications and memory of indoor environments. GOAT-Bench provides an open-vocabulary, multi-modal lifelong benchmark and compares methods’ performance and memory use.
1. Introduction
- The benchmark addresses the limits of single-modality goals: images may be impractical to provide for every object, while category names may not distinguish specific instances.A plate category alone, for example, does not specify a plate with a red pattern.Source: §1 · p. 2
- GOAT-Bench is a reproducible benchmark for multi-modal lifelong navigation, built from 181 HM3DSem scenes, 312 object categories, and 680k episodes.It evaluates a sequence of goals specified by category, image, or language description.Source: §1 · p. 2
- Unlike benchmarks that reset the scene after each target, GOAT-Bench episodes contain 5 to 10 targets across distinct modalities and support open-vocabulary targets, including those unseen during training.The cited prior benchmarks are described as often limited to 6 to 21 categories.Source: §1 · p. 2
- SenseAct-NN methods achieve 2.9–4.6% higher success rates than modular methods, but 4.7–9.2% worse SPL.The comparison attributes modular methods’ efficiency to their use of semantic maps and identifies effective memory representations for SenseAct-NN as a future research direction.Source: §1 · p. 2
- Memory improves efficiency for both method classes on later episode subtasks, while the methods perform poorly on language and image goals, particularly with CLIP features.With memory, later-stage SPL improves by approximately 1.9x for SenseAct-NN and 1.5x for modular methods; SenseAct-NN methods are more robust to the specified goal noise than modular methods.Source: §1 · p. 2
2. Related work
Prior navigation work commonly evaluates single goal types with modality-specific components, while multi-modal embodied tasks often remain episodic. GOAT-Bench instead studies open-vocabulary, multi-modal navigation over lifelong sequences and offers a reproducible complement to real-world evaluation.
2. Related work
- Many navigation benchmarks assess individual goal types with modality-specific encoders or recognition modules, which do not generalize across modalities out of the box.Prior work using CLIP to bridge image and object goals does not focus on longer descriptions for instance disambiguation; the paper reports that CLIP encodings do not help with specific object instances.Source: §2 · p. 3
- Multi-modal embodied tasks such as ALFRED and EmbodiedQA use multiple input modalities but are studied in single-episode settings; GOAT-Bench focuses on lifelong, open-vocabulary navigation goals.A related multi-object navigation task uses a closed set of object categories, whereas GOAT goals span object category, image, and language description.Source: §2 · p. 3
- Concurrent benchmarks and systems differ from GOAT-Bench in goal sequence, modality, or vocabulary scope, while its reproducible simulation benchmark is intended to complement rather than replace real-world benchmarking.The paper says reproducibility allows analysis of memory and robustness questions that remain unanswered in time-intensive real-world evaluations.Source: §2 · p. 3
3. Task
GOAT tasks ask an agent to navigate sequentially to 5 to 10 goals in an unseen indoor environment, with each goal specified by category, language, or image. The setting is open-vocabulary and continues across goals rather than resetting the scene after each success.
3. Task
- Each GOAT episode contains 5 to 10 subtasks, each defined as finding a goal specified by category name, language description, or an identifying image.The agent is spawned randomly in an unseen indoor environment.Source: §3 · p. 3
- The task is open-vocabulary, and after reaching a goal or exhausting its time budget, the agent receives the next goal without a scene reset.Chaining multi-modal goals enables evaluation of methods that leverage experience from earlier in the same environment.Source: §3 · p. 3
- The agent uses a Stretch embodiment with RGB, depth, and relative-pose inputs; a subtask succeeds when it stops within 1 m of the goal within 500 actions.Actions include moving forward by 0.25 m, turning or looking by 30 degrees, and stopping.Source: §3 · p. 3
4. Dataset
GOAT-Bench combines open-vocabulary object, image, and language goals in lifelong episodes, with goals and episodes generated across multiple scenes and evaluation splits.
4. Dataset
- Figure 2 previews goal instances as object images, language descriptions, and object-category annotations.The figure uses blue for images, orange for descriptions, and green for category annotations.Source: Figure 2 · p. 4
- GOAT-Bench covers 264 training categories, ∼13k goal specifications, and 725k training episodes, with about 9x/6x the training/validation categories of prior closed-set datasets.The episodes cover 145 training scenes, with 5k episodes per scene.Source: §4 · pp. 3–4
- OVIIN extends image-goal navigation beyond the canonical six categories, providing 7.7k training instances across 264 categories and 2.9k validation instances across 164 categories.The splits use the same heuristics as OVON to support generalization to novel instances and categories.Source: §4 · p. 4
- LanguageNav generates descriptions by combining simulator-derived spatial and semantic context with BLIP-2 appearance attributes, then prompting ChatGPT-3.5 to describe each goal.The pipeline yields 5.4k training goal instances from 225 categories and 1.9k instances from 137 held-out categories.Source: §4 · p. 5
- Episodes contain 5–10 goals across category, description, and image modalities, sampled uniformly by modality and goal instance.Starting positions are on the same floor as all goal locations, and the first goal is 1–30 m away.Source: §4 · p. 5
- Evaluation uses VAL SEEN, VAL SEEN SYNONYMS, and VAL UNSEEN splits to test generalization to new environments and category familiarity.The synonym split evaluates categories such as “sofa” after training on “couch.”Source: §4 · p. 5
5. Baselines
The baselines contrast modular navigation with sensor-to-action policies trained by reinforcement learning, including task-specific skill chains and a monolithic policy intended to retain experience across subtasks.
5. Baselines
- The benchmark compares semantic-mapping and planning-based modular methods with SenseAct-NN policies trained using reinforcement learning.The policy comparison includes versions with and without implicit memory.Source: §5 · p. 5
5.1. Modular Baseline
- Modular GOAT chains detection, exploration, and last-mile navigation modules, using semantic and instance maps to match image or language goals to explored objects.Image goals use keypoint matching; language goals use cosine similarity between CLIP features, while category information filters candidate instances.Source: §5.1 · p. 5
- CoW uses CLIP features to match image, object, and language goals against images seen during exploration.Unlike Modular GOAT, it does not use the described instance-specific memory and category filtering.Source: §5.1 · p. 6
5.2. SenseAct-NN Baselines
- SenseAct-NN Skill Chain trains separate policies for ObjectNav, Instance ImageNav, and LanguageNav, with a high-level planner selecting the policy by goal modality.The policies use CNN+RNN architectures, and their encoders vary by subtask; the image-goal policy uses CroCo-v2 embeddings.Source: §5.2 · p. 6
- The monolithic SenseAct-NN policy uses CLIP text and image encoders for multimodal goals and carries its recurrent hidden state across subtasks as implicit memory.It is trained end-to-end with reinforcement learning on the GOAT training dataset.Source: §5.2 · p. 6
6. Results
SenseAct-NN Skill Chain leads the non-ground-truth baselines in success across evaluation splits, while Modular GOAT achieves higher SPL. The results also identify weak monolithic performance and a substantial perception-related gap for modular methods.
6.1. Modular vs. SenseAct-NN approaches
- Modular GOAT achieves 6.6% higher SPL than SenseAct-NN Skill Chain and 10.9% higher SPL than SenseAct-NN Monolithic, using explicit semantic and instance maps.The instance memory supports localizing goals in previously explored parts of the map and navigating to them directly.Source: §6.1 · p. 6, Table 2 · p. 7
- About 4% higher average success than modular baselines across all splits: SenseAct-NN Skill Chain leads non-ground-truth baselines, but averages 6.6% lower SPL than best Modular GOAT.Table 2 compares end-to-end RL and modular methods on VAL SEEN, VAL SEEN SYNONYMS, and VAL UNSEEN.Source: §6.1 · p. 6, Table 2 · p. 7
- Ground-truth semantics improve Modular GOAT by an average of ∼30% in success and ∼22.0% in SPL.This comparison separates perception limitations in object detection and map building from matching, planning, and last-mile navigation.Source: Table 2 · p. 7
- SenseAct-NN Monolithic performs worse than the other baselines; the authors hypothesize that CLIP’s instance-feature limits and long-horizon RL difficulty contribute.The policy performs particularly poorly on image and language goals.Source: §6.1 · p. 7
7. Analysis
The analysis finds substantial modality-specific differences, memory-dependent efficiency effects, and varying gains over sequential subtasks; robustness also varies by method and goal modality.
7.1. How do agents perform on each modality?
- For object goals, Modular GOAT reaches 29.4% success versus 25.8% for SenseAct-NN Skill Chain and 25.7% for Monolithic; for language goals, it beats Skill Chain by about 5% success and more than 2x SPL.The analysis breaks out baseline performance by object, language, and image goals.Source: §7.1 · p. 7, Figure 4 · p. 7
- For image goals, CroCo-v2 Instance ImageNav in SenseAct-NN Skill Chain exceeds Modular GOAT by about 15% in success, while Modular GOAT has better SPL; Monolithic underperforms on both.Average baseline rankings do not reflect performance across individual modalities.Source: §7.1 · p. 7, Figure 4 · p. 7
7.2. How important is memory for efficient navigation?
- Removing memory lowers Modular GOAT SPL from 17.6 to 9.4 (about 2x) and success from 26.4 to 21.2.The memory ablation rebuilds the map from scratch for each subtask.Source: §7.2 · p. 7, Figure 5 · p. 8
- When memory is dropped, SenseAct-NN Monolithic shows only minor declines: SPL falls from 9.4 to 9.0 and success from 16.8 to 14.9, suggesting its hidden state captures little useful explored-scene information.Source: §7.2 · p. 7
7.3. Does success and efficiency improve over time?
- Across subtasks, Modular GOAT success does not improve, while its SPL rises from 12.4 to 18.7 over the first three subtasks before saturating at 18.4.The analysis plots success and SPL against the number of completed subtasks.Source: §7.3 · p. 8, Figure 6 · p. 8
- SenseAct-NN Monolithic improves over time from 5.6 to 10.6 SPL and from 10.6% to 20.0% success.The reported gains occur as more subtasks are completed in the same episode.Source: §7.3 · p. 8, Figure 6 · p. 8
7.4. How robust are these methods to noise in goal specifications?
- With image noise, category synonyms, and paraphrased language as perturbations, SenseAct-NN Skill Chain is most robust (25% average success drop) and Modular GOAT least robust (53% drop).The evaluation uses the VAL SEEN split.Source: §7.4 · p. 8, Figure 7 · p. 8
- Modular GOAT is especially affected by category synonyms because DETIC detects uncommon synonyms poorly; all baselines lose language-goal success, while SenseAct-NN methods show little image-goal loss under Gaussian noise.The authors attribute the language-goal weakness to limited instance-specific expressiveness in CLIP goal embeddings.Source: §7.4 · p. 8
8. Conclusion
GOAT-Bench is a reproducible benchmark for multimodal lifelong navigation. The authors report that effective memory representations improve task efficiency and analyze memory dependence, modality performance, and robustness to goal noise.
8. Conclusion
- GOAT-Bench benchmarks modular and end-to-end methods with and without memory for multimodal lifelong navigation, and the authors report that effective memory representations improve efficiency.The paper also analyzes memory dependence, performance across modalities, and robustness to goal-specification noise.Source: §8 · p. 8
A. Open-Vocab Instance-ImageNav dataset
The Open-Vocab Instance ImageNav dataset expands image-goal diversity by sampling views around object instances and filtering goals for visibility.
A. Open-Vocab Instance-ImageNav dataset
- The dataset uses open-vocabulary object categories and instances, captures candidate-view images around each instance, then filters image goals using frame- and object-coverage heuristics.It builds on the pipeline in [24] and uses categories and instances from rather than only the six canonical ObjectNav categories.Source: A. Open-Vocab Instance-ImageNav dataset · p. 12
B. LanguageNav Dataset
The LanguageNav dataset generates target descriptions from selected viewpoints, combining image captions with nearby-object semantics and metadata. The appendix also outlines SenseAct-NN skill-chain and monolithic baseline architectures.
B. LanguageNav Dataset
- For LanguageNav, the authors sample viewpoints around OVON goal instances, capture 512 × 512 images, filter for target visibility, and use the highest-coverage view for BLIP-2 description generation.Candidate viewpoints are sampled at radii 1.0, 1.5, or 2.0 meters in 10-degree sections around the object centroid.Source: B. LanguageNav Dataset · p. 12
- They select nearby objects at least 4.5 m away with over 0.5% frame coverage, then use ground-truth semantics and depth to assemble metadata for caption generation.The metadata includes bounding boxes, area coverage, category, semantic index, and whether an object is the target.Source: B. LanguageNav Dataset · p. 12
- The pipeline combines BLIP-2 captions and bounding-box metadata in a GPT-3.5-Turbo prompt, storing the generated response as the language goal description.Additional dataset examples are shown in Figure 11.Source: B. LanguageNav Dataset · p. 12
C. Baselines
- SenseAct-NN Skill Chain selects a goal-specific pretrained policy and switches policies after each target; Monolithic uses one policy across goal types while maintaining hidden state.Both concatenate goal and current-visual-observation embeddings before predicting actions.Source: Figure 8 · p. 12, C. Baselines · p. 12
C.1. SenseAct-NN Skill Chain
The skill-chain baseline trains separate navigation policies for different goal modalities, while the monolithic and modular baselines offer alternative GOAT architectures.
C.1. SenseAct-NN Skill Chain
- The skill chain trains separate ObjectNav, LanguageNav, and Instance ImageNav policies, combining visual and pose features with modality-specific goal embeddings in a GRU.The policies use different goal encoders, including CLIP for object goals, BERT for language, and CroCo-v2 for instance images.Source: §C.1 · pp. 12–14
- Language instructions are generated from region semantics and target-object captions, describing location with relative directions rather than absolute coordinates.Source: §C.1 · p. 14
- InstanceImageNav performs best across the three individual tasks, while LanguageNav struggles most with instance-specific information.The authors attribute InstanceImageNav’s performance to CroCo-v2’s cross-view-consistent goal embeddings and LanguageNav’s difficulty to CLIP features inadequately capturing instance-specific information.Source: §C.1 · p. 14
C.2. SenseAct-NN Monolithic Policy
- The monolithic policy encodes RGB, GPS+Compass, and modality-specific goals, then carries its GRU hidden state between subtasks as implicit memory.Source: §C.2 · p. 14
C.3. Modular GOAT
- The modular baseline builds a semantic and instance-specific top-down map, matches object, language, or image goals, and passes found goals to a local policy.It explores with frontier exploration until a match is found.Source: Figure 10 · p. 15, §C.3 · p. 15
D. Qualitative Analysis of SenseAct-NN Monolithic Agent
A qualitative episode illustrates that the monolithic policy’s hidden state did not reliably preserve information about previously seen objects and regions.
D. Qualitative Analysis of SenseAct-NN Monolithic Agent
- After seeing a bathroom cabinet en route to a washer dryer and completing an oven-and-stove goal, the agent wanders instead of returning to the cabinet.The example highlights the current GRU hidden state’s limited effectiveness at tracking seen objects and regions.Source: D. Qualitative Analysis of SenseAct-NN… · p. 15