Source-linked AI summary
Vesta: A Generalist Embodied Reasoning Model
Johan Bjorck, Zhiqi Li, Yunze Man, Jing Wang, An-Chieh Cheng, Sifei Liu, Shihao Wang, Zhiding Yu, Abhishek Badki, Stan Birchfield, Valts Blukis, Yevgen Chebotar, Siyi Chen, Sicong Leng, Yu-Cheng Chou, Tianli Ding, Boyi Li, Zhengyi Luo, Hang Su, Jonathan Tremblay, Tingwu Wang, Bowen Wen, Jimmy Wu, Xianghui Xie, Hanrong Ye, Hongxu Yin, K. R. Zentner, Liangyan Gui, Yu-Xiong Wang, Yuke Zhu, Linxi "Jim" Fan, Jan Kautz
TL;DR
Robots must integrate localization, spatial reasoning, navigation, memory, and long-horizon planning, but specialist stacks are costly and difficult to scale. Vesta unifies these capabilities with curated spatially grounded training data and a multimodal memory harness, outperforming individual and ensemble baselines across benchmarks and improving real-robot task success.
Problem
Specialist models address embodied capabilities in isolation, leaving evidence for a scalable unified alternative limited despite the need to integrate them.
Method
Vesta combines a curated corpus for spatially grounded capabilities with a multimodal memory harness that conditions planning on prior observations and subtasks.
Results
38.3% higher success on memory-heavy real-world tasks accompanies average benchmark gains of >20 points over the strongest individual baseline and >10 points over an oracle ensemble.
Takeaways & Limitations
A single generalist planner can match or exceed specialists across four capability axes while offering a feasible, scalable alternative to assembling specialist models.
Takeaways & Limitations
Real-robot evaluation covers one bimanual platform and three memory- and reasoning-heavy tasks, leaving broader embodiments and planner-actor interfaces untested.
Abstract
from arXiv · showhide
Robots operating in open-world environments must seamlessly integrate localization, spatial reasoning, navigation, and long-horizon planning. While specialist models excel at individual tasks, deploying a multi-model stack is computationally expensive and prone to cascading errors. We present Vesta, a unified embodied generalist that consolidates these capabilities into a single foundation model. Our approach combines a diverse and massive curated corpus designed to induce spatial grounding and a simple multimodal memory harness that enables reasoning over extended time horizons. Across diverse benchmarks, Vesta on average beats individual SOTA baselines by >$20\%$ and beats an ensemble of per-category-best baselines by $>10\%$ -- thus demonstrating that a generalist model can match or exceed specialists. On real-world robotic tasks requiring memory and reasoning, Vesta improves task success by >35\%. Our work thus demonstrates that a single generalist is a feasible, scalable, and arguably preferable alternative to combining specialists.
1. Introduction
Vesta unifies localization, navigation, embodied reasoning, and action planning in a single generalist model, addressing the scalability and cascading-error risks of specialist ensembles. A curated SFT corpus and simple memory harness enable strong cross-task performance and 38.3% higher success on memory- and long-horizon reasoning tasks on real robots.
- Motivation: Specialist models excel in isolated capabilities but are difficult to deploy together because multi-model ensembles are not scalable and can propagate errors.The literature commonly develops navigation, memory, and reasoning specialists in separate silos, despite their strong in-domain performance.
- Approach: Vesta unifies localization, navigation, embodied reasoning, and action planning into one generalist embodied model.The model supports multimodal inputs and hierarchical control across four main capabilities.
- Approach: Vesta combines a curated supervised fine-tuning corpus with a simple multimodal memory harness to support grounding, navigation, reasoning, and real-robot data.The training recipe is designed to improve cross-task generalization compared with naive supervised fine-tuning.
- Results: >20% above the strongest single baseline and >10% over an oracle ensemble across four capabilities, Vesta demonstrates that one generalist can unify them without hurting benchmark scores.The oracle ensemble uses the best baseline in each individual category.
- Results: 38.3% higher average task success on real-robot tasks requiring memory and long-horizon reasoning shows Vesta’s benefit beyond benchmarks.The empirical validation uses a bimanual robot platform and memory-intensive tasks.
2. Methods
Vesta is finetuned from Qwen3-VL-8B with supervised training across localization, navigation, embodied reasoning, and memory-conditioned planning. Its methods combine curated spatial-grounding data, high-level VLN action prediction, and an explicit memory harness for long-horizon planning.
- Training framework: Vesta is finetuned from the Qwen3-VL-8B base model using SFT for localization, navigation, embodied reasoning, and memory-conditioned planning.The method emphasizes data curation to build these capabilities.
- Localization and grounding: Grounding associates text descriptions with spatial regions and predicts contact or manipulation points, connecting perception to action.The localization dataset uses a base–tail strategy, with Objects365, COCO, and LVIS providing broad category coverage and dense annotations.
- Vision-and-Language Navigation: VLN conditions high-level decisions on a route instruction, current egocentric view, and sampled visual history within the standard R2R-style formulation.Low-level motion is handled by a navigation backend, while the model predicts pixel goals, turn sequences, or related navigation actions.
- Embodied reasoning: Embodied reasoning extends localization to action-conditioned scene understanding, including affordance and placement prediction, manipulation waypoints, and task progress estimation.The training corpus mirrors a hierarchy from general visual understanding to embodied tasks.
- Long-horizon planning: The memory harness curates prior observations and predicted subtasks into a bounded context that is re-injected into each planning prompt.Historical images are capped at K using uniform or recency-biased sampling, addressing the non-Markovian nature of long-horizon planning.
3. Training recipe
Vesta’s SFT corpus is intentionally biased toward spatially grounded capabilities. Spatial intelligence is the largest component, complemented by navigation, grounding, general VLM data, and embodied reasoning or real-robot data.
- Corpus composition: 27.1% of SFT samples target spatial intelligence, the corpus’s largest category.The corpus is intentionally biased toward spatially grounded capabilities.
- Corpus composition: 21.8% of samples support navigation, while 20.8% support grounding.Together, these categories form another 42.6% of the corpus.
- Corpus composition: 16.2% of samples are general VLM data to preserve broad visual-language competence and reduce over-specialization.The remaining data consists of embodied reasoning and real robot data for task-level reasoning.
4. Evaluation
Vesta achieves strong performance across embodied cognition and localization benchmarks, ties the SOTA navigation specialist, and improves real-robot planning over actor-only and Qwen3-VL-planner baselines. Evaluation uses continuous offline rollouts, temporal IoU scoring, and held-out navigation episodes to assess long-horizon embodied reasoning.
- Embodied benchmarks: Vesta achieves the highest average score in both embodied cognition and localization, leading on most benchmarks and remaining competitive on the rest.It also produces a clear average improvement over the existing 8B model.
- Offline action planning: The offline planning benchmark evaluates dynamically selected subtasks through continuous temporal rollouts rather than isolated-frame predictions.At each fixed time step, the planner receives the current observation, semantic goal, and history, then selects the next subtask from generated candidates.
- Offline action planning: Temporal Intersection-over-Union weights predictions by the duration of overlapping ground-truth segments and preserves the planner ordering observed on real-robot tasks.Overlap is counted only when the predicted action matches the ground-truth label.
- Navigation: Vesta ties the SOTA navigation specialist InternVLA-N1, leading on SR and OS while trailing on SPL and NE across 1839 held-out navigation episodes.Evaluation uses the R2R val_unseen split in Habitat, with scenes and episodes excluded from supervised fine-tuning data.
- Real-robot manipulation: 38.3%: Vesta improves average real-robot success over the actor-only baseline, and 25% over the Qwen3-VL planner.The study uses Gr00t-N1.6 as the actor, evaluates three tasks and configurations, and tests each task with 20 samples.
- Ablations: The unified data mixture, transition-step oversampling, and combined image-text memory each improve evaluation performance relative to their corresponding alternatives.Image-only memory loses progress awareness, text-only memory encourages history shortcuts, and 2× transition oversampling outperforms the default while 3× adds only marginal improvement.
5. Related Work
Related work has advanced embodied control through vision-language models, specialized navigation systems, and memory mechanisms, but many approaches remain modular or lack explicit history. Vesta is positioned against this specialist-stack paradigm as a unified embodied generalist.
- Embodied Vision-Language Models: Embodied control increasingly uses pretrained vision-language models as robotic brains, with reinforcement-learning reasoning and chain-of-thought methods becoming common.These developments have helped establish VLM-based reasoning as a major direction for embodied control.
- Embodied Vision-Language Models: Modern embodied systems commonly combine a planner VLM with an actor VLA, while alternative approaches use multiple specialist models for different applications instead of one generalist checkpoint.The passage contrasts this planner–actor architecture and specialist-model strategy with generalist modeling.
- Vision-Language Navigation: Vision-language navigation has progressed from map- and SLAM-based methods toward foundation-model systems, with modern specialists typically produced by fine-tuning VLMs and evaluated in simulation.Earlier navigation methods used precomputed or geometric maps, depth sensors, or monocular-camera SLAM.
- Memory in Robotics: Long-horizon robotic tasks require memory, yet many VLAs lack explicit history; proposed solutions include spatial maps, visual traces, keyframes, experience retrieval, and language-based abstractions.Related methods also pair short-horizon video with long-horizon language tracking or abstract events into language.
6. Discussion
Vesta is a generalist embodied planner that matches or surpasses domain specialists across four capability axes. It also exceeds strong baselines and improves memory-heavy real-world manipulation success.
- Vesta matches or beats domain specialists simultaneously across four capability axes.
- >20 points above the strongest individual baseline across the four capabilities.
- >10 points above an ensemble of baselines across the four capabilities.The ensemble takes the strongest model in each category.
- 38.3% higher success rate on memory-heavy real-world manipulation tasks over an actor-only baseline.
A. Real Robot Evaluation · B. Real-Robot Inference Loop
Real-robot evaluation uses a bimanual YAM-gripper platform with Vesta’s planner VLM issuing natural-language commands to a Gr00t-N1.6 VLA actor. The planner and actor communicate through a shared natural-language subtask under synchronous or asynchronous coupling, while planner memory remains private.
- A. Real Robot Evaluation: Figure 8 depicts hierarchical execution: the planner continuously observes images, stores text and images in memory, and sends subtasks to an actor that produces robot actions.The actor consumes images, states, and subtasks; this work focuses on improving the planner.
- A. Real Robot Evaluation: Real-robot task details, including per-task sample counts and success criteria, are provided in Table 6.The main paper reports all evaluation numbers using synchronous actor–planner coupling.
- A. Real Robot Evaluation: The robotic platform uses bimanual YAM grippers from I2RT robotics, with the planner VLM sending text commands to the actor model.This follows the standard planner–actor setup illustrated in Figure 8.
- B. Real-Robot Inference Loop: The deployment couples Vesta’s high-level planner πplan with a Gr00t-N1.6 VLA low-level actor πact through the current natural-language subtask z.The actor maps observation o_t and z to action a_t without scene history or the planner’s reasoning trace.
- B. Real-Robot Inference Loop: Both coupling modes share the same control-loop body and reset semantics, differing only in when fresh planner inference may update the current subtask.The modes are summarized side-by-side in Algorithms 1 and 2.
- B. Real-Robot Inference Loop: In both modes, the actor publishes its latest observation to the planner and the planner publishes its latest subtask to the actor, while planner memory remains on the planner side.The shared freshness contract requires that the actor never execute with a subtask older than τmax.
C. More on Training Details
Vesta is trained through full-parameter supervised fine-tuning of Qwen3-VL-8B-Instruct, while a separately fine-tuned Gr00t-N1.6 VLA serves as the low-level actor. The model components are jointly optimized, and the actor is trained on demonstrations for language following and motion smoothness.
- More on Training Details: The actor and planner interact only at inference time through the natural-language subtask described in Section B.The actor is separately fine-tuned for the real-robot evaluation.
- Supervised Fine-Tuning: Vesta uses full-parameter supervised fine-tuning from Qwen3-VL-8B-Instruct on the data mixture described in Section 3.The SFT hyperparameters are listed in Table 7.
- Supervised Fine-Tuning: The vision tower, multi-modal projector, and language backbone are unfrozen and trained jointly in pure bf16.Optimization uses fully sharded data parallelism, FlashAttention-2 kernels, and sequence packing for long-context multi-image samples.
- Low-level actor VLA training: The low-level actor is Gr00t-N1.6, a Vision-Language-Action model fine-tuned on collected demonstrations.The training aims to support language following and motion smoothness, with hyperparameters listed in Table 8.
D. Related Work
Related work advances embodied foundation models with specialized mechanisms for physical-space dynamics, while exposing persistent challenges in multi-stage reasoning, cross-task adaptation, and long-horizon logic. Researchers address these challenges through new benchmarks, hierarchical abstractions, and multimodal memory systems for partially observable environments.
- Embodied Foundation Models: Embodied foundation models translate web-scale semantic knowledge into low-level robotic control and spatial reasoning.Recent systems incorporate continuous flow-matching, 3D representations, and state-space modeling for continuous physical dynamics.
- Embodied Foundation Models: Models still struggle with multi-stage reasoning, cross-task adaptability, and long-horizon logic, motivating hierarchical abstractions and explicit multimodal memory modules.These approaches target limitations that remain despite structural advances in embodied foundation models.
- Benchmarking and Evaluation of Robotics Foundation Models: Spatial-intelligence and embodied question-answering benchmarks evaluate visual perception, logical deduction, and spatial grounding across egocentric and multi-view modalities.The evaluation landscape also increasingly diagnoses robotic failure modes, shortcut biases, and vulnerabilities.
- Benchmarking and Evaluation of Robotics Foundation Models: Because partially observable environments require memory for ambiguity resolution and long-horizon tasks, researchers develop multimodal, multi-scale memory to avoid costly dense frame histories.Examples combine short-horizon dense video encoding with long-horizon language-based event tracking, or abstract past events into natural language.
E. Limitations
Vesta’s limitations concern the narrow scope of real-robot evaluation, uncertain scaling beyond the 8B model, and a memory harness that remains largely hand-designed. Future work must test broader embodiments, study scaling and deployment, and learn memory policies for lifelong operation.
- Real-robot evaluation scope: Real-robot evaluation covers one bimanual YAM gripper platform and three memory- and reasoning-heavy tasks.The setup assumes natural-language subtasks can be faithfully executed at a roughly fixed control rate.
- Real-robot evaluation scope: Future evaluations should stress planner–actor interfaces in humanoids and mobile manipulators with coupled locomotion, manipulation, and navigation.These embodiments challenge assumptions in the current setup and motivate adapting subtask abstractions to different actor controllability profiles.
- Scaling, distillation, and edge deployment: 8B is the only reported scale: Vesta is built on Qwen3-VL-8B, while scaling behavior for embodied SFT mixtures remains poorly understood.Those mixtures are dominated by long-tail spatial and trajectory data, unlike the web data used to characterize most known VLM scaling effects.
- Learned and lifelong memory: Vesta’s memory harness retains fixed frames through uniform or recency-biased sampling and records textual subtasks, rather than learning memory selection.Lifelong operation raises when to consolidate, forget, or re-retrieve information across episodes in the same home or warehouse.
F. Broader Impacts
Vesta’s generalist planning could simplify embodied-robot deployment and research by replacing brittle specialist stacks, while raising safety, labor, and compute-concentration concerns. Its hierarchical planner/actor design and controlled, supervised evaluation provide mitigation surfaces before future broader release.
- Potential positive impacts: Generalist planners could simplify assistive-robot deployment across domestic, healthcare, retail, and warehouse settings while lowering barriers for academic and small-team robotics research.Reducing separate models also reduces failure points and makes system behavior easier to audit.
- Potential negative impacts: Improved high-level planning with capable low-level controllers raises physical-safety risks from incorrect actions in shared spaces and possible labor displacement in routine physical tasks.These concerns are identified as standard risks associated with embodied AI.
- Potential negative impacts: Training and serving large VLM-based planners is compute-intensive, potentially concentrating capability in well-resourced organizations.The compute burden is presented as a potential negative impact of Vesta-like systems.
- Mitigation considerations: Vesta’s hierarchical planner/actor split enables safety filters on textual subtasks before motor commands and sandboxing to a vetted action vocabulary.The authors restrict evaluation to a controlled lab environment with researcher supervision and plan documentation for future asset releases.