Source-linked AI summary
Deploying Foundation Models for Embodied Navigation
Vishnu Sashank Dorbala, Dinesh Manocha
TL;DR
Foundation models can improve zero-shot embodied navigation, but deployment is limited by compute, context length, and poor personalization in novel environments. The paper presents TAP, which primes FMs with learned object-transit behavior, and MemCtrl, which filters observations through an active memory head. It reports that MemCtrl improves performance of low-parameter FMs while using less context.
Problem
Method
TAP primes FMs with scene-specific transit behavior learned from human habits, while MemCtrl adds an active memory head that filters useful observations.
Source: §4.1 · pp. 6, 8, §4.2 · p. 8Results
MemCtrl improves the performance of low-parameter FMs with lower context size.
Source: §5 · p. 13Takeaways & Limitations
The paper presents TAP and MemCtrl as contributions toward enabling real-world deployment of FM-based embodied agents.
Source: §5 · pp. 12–13Perceiver methods face high inference latency and weak spatial reasoning, planners can hallucinate nonsensical plans, and policies require massive data and may be brittle out of distribution.
Source: §3 · p. 5
Abstract
from arXiv · showhide
We present and tackle two problems associated with deploying Foundation Models (FMs) on Embodied Agents performing navigation: 1) Training bias in FMs leading to poor personalization in unseen environments, and 2) Limited FM context length hindering success, especially on long horizon tasks. Our solution for the former involves priming the FM with human-habit data mined from the scene and our solution for the latter involves active memory management via a novel `memory head' augmentation. We first present a taxonomy of existing literature on FM-based Embodied Navigation, and highlight these limitations. We then present our approaches, Transit-Aware Planning (TAP) and MemCtrl to address the limitations. With TAP, we present real-world results in a lab environment with a Turtlebot for personalized target finding that shows an average improvement of 18% over a non-TAP baseline. On MemCtrl, we report a 6% average improvement across various embodied tasks, with 20% on long instruction subsets, all while using nearly half the context used in the baseline model. Motivated by these result, we present our stance the deployability of FM-based embodied agents in real-world environments, and highlight open research directions.
1 Introduction
Embodied agents are expected to perform varied human-directed tasks with little decision-making effort, but deploying foundation models brings compute, context-length, and personalization challenges. The paper introduces TAP and MemCtrl to address training and inference shortcomings and improve zero-shot deployability.
1 Introduction
- Generalist embodied agents aim to perform a wide range of human-described tasks with minimal or no human effort in decision-making.Objectives may be described with text, images, or sounds, and include tasks such as finding objects, folding clothes, or cooking.Source: §1 · p. 1
- FMs can improve generalist capabilities, but fixed context size and poor personalization to novel environments limit their deployment.The paper also notes that simulator-trained models may require on-device compute that real robots often lack.Source: §1 · p. 1
- TAP targets personalized path planning in dynamic scenes, while MemCtrl adds a memory head to improve FM memory efficiency.The paper also presents a taxonomy of FM-based navigation approaches and discusses future research directions for deployment in human-centric environments.Source: §1 · p. 2
2 Background
Embodied navigation includes tasks such as finding targets in unfamiliar environments, sometimes with little guidance or no map. Foundation models have improved zero-shot performance, but deployment remains constrained by context, memory, and the availability of task-relevant context.
2 Background
- Navigation tasks include finding an object in an unfamiliar environment before performing a related manipulation, often without a map or detailed guidance.The paper describes this as requiring decisions about unknown layouts and visual grounding of the target; it contrasts navigation with manipulation challenges such as estimating material properties and modeling contact.Source: §2 · p. 2
- FMs used in planning systems or end-to-end schemes have shown large performance improvements in zero-shot settings, yet deployment can fail when task-relevant context is absent from inputs, representations, or observation history.Limited context length, poor interpretability, and memory-inefficient architectures also hinder deployment.Source: §2 · p. 3
3 FM-Based Methods for Embodied Navigation
The paper organizes FM-based embodied navigation into perceiver, task-planner, and policy approaches. It outlines distinct deployment limitations for each category.
3 FM-Based Methods for Embodied Navigation
- FM perceivers turn observations into semantic descriptions for downstream control, task planners generate subgoals for low-level execution, and policies map perception and instructions directly to actions.The taxonomy distinguishes where the FM sits in the navigation pipeline: before planning, in place of task planning, or across perception, planning, and control.Source: Figure 1 · p. 3, §3 · pp. 3–4
- Perceiver methods use FMs as visual-semantic modules whose outputs feed a downstream control system.The section describes examples that use language models or vision-language models for commonsense reasoning, landmark grounding, or frontier scoring, often without fine-tuning.Source: §3 · p. 4
- Task-planner methods decompose navigation goals into high-level subgoals, which a separate low-level controller grounds and executes.The section also describes using environment affordances to assess plan feasibility and a benchmark testing combinations of language and vision-language models.Source: §3 · p. 4
- Policy methods translate raw perception and instructions into navigation actions, using robot-data training from scratch or fine-tuning an internet-trained FM.The section cites RT-1 and RT-2 as examples of these two approaches.Source: §3 · p. 4
- FM perceivers can incur high inference latency and weak spatial reasoning, planners can produce infeasible plans that are costly to replan, and policies require massive training data and risk brittle out-of-distribution performance.The paper notes that small camera-angle changes can break policy outputs.Source: §3 · p. 5
4 Challenges with Deploying FM-Based Embodied Agents
Deployment is challenged by simulator training bias and limited context, especially for sequential embodied decisions. The section presents TAP for personalized object-transit planning and MemCtrl for active memory filtering, with reported gains on long-horizon and complex tasks.
4 Challenges with Deploying FM-Based Embodied Agents
- Limited context and simulator training bias constrain FM deployment, while sequential embodied tasks require deciding which observations to retain.The paper identifies simulator mismatch as a source of out-of-distribution bias and limited context as an inference challenge.Source: §4 · pp. 5–6
4.1 Personalization: Modelling Portable Targets
- TAP frames portable ObjectNav as learning human placement habits and using modeled object transits for temporally aware planning.Dynamic Object Maps convert topological scene graphs into dynamic node-attributed graphs, and a learned transit model primes the FM.Source: §4.1 · pp. 6–8
- TAP-LLM improves success over a vanilla LLM in 50 lab episodes, with transit awareness helping when routine paths have structure.The experiment uses a simulated lab environment transferred to real-world personalized assistance.Source: Figure 3 · p. 7, Table 2 · p. 8
4.2 Memory Efficient FMs
- MemCtrl adds a trainable memory head to filter observations on the fly, keeping or discarding them atop a frozen MLLM backbone.The head is designed for real-time filtering in memory-constrained settings involving small models.Source: §4.2 · pp. 8–9, Figure 4 · p. 8
- MemCtrl uses less memory and achieves better performance than the Memo-style baseline that stores the full history as context.The context-growth comparison uses EB-Habitat, Qwen2.5-VL-7B-Ins, and an offline-supervised memory head trained with GPT-4o data.Source: Figure 6 · p. 10, §4.2 · p. 10
- 6% average improvement and nearly 20% on long-horizon instructions are reported for MemCtrl across EB-ALFRED and EB-Habitat subsets.Table 3 evaluates three MemCtrl variations on five subsets across three trials.Source: §4.2 · p. 10, Table 3 · p. 10
- Any memory-head variant improves performance across settings, including long-form and complex instructions with irrelevant information.The paper reports overall performance improvement on complex instructions as well as long-form instructions.Source: §4.2 · pp. 10–11
- On a long-horizon plate task, both memory-head agents continue after GPT-4o stops early; µRL explores, while µExpert repeats the same activity.The authors identify investigation of these distinct behaviors as future work.Source: §4.2 · p. 12, Figure 8 · p. 12
5 Conclusion & Future Directions
The conclusion presents two approaches to deployment challenges: priming for portable object finding and a memory head for inference-time limits. It frames day-one unboxing as a way to ask what an agent should do reliably at deployment and what it should learn from scene context.
5 Conclusion & Future Directions
- The paper addresses training bias in portable object finding by priming FMs with learned dynamic-object behaviours, and inference limits with MemCtrl, a memory-head architecture.Source: §5 · pp. 12–13
- Day-one unboxing asks how to estimate whether an agent can perform reliably in an unseen environment before deployment.Source: §5 · p. 13
- The authors ask which core functions should be reliable before deployment and which should instead be acquired from scene context, favouring critical skills over training for every task.Source: §5 · p. 13