PaperlayerView exact revision on arXiv

Source-linked AI summary

AtlasVLA: Persistent World-Ego State Modeling for Vision-Language-Action Models

Guiyu Zhao, Longteng Guo, Yanghong Mei, Zilin Zhu, Yu Zhang, Bin Cao, Mingming Yu, Xingjian He, Jie Jiang, Jing Liu

arXiv:2608.06729v1cs.ROcs.CV

TL;DR

Reactive vision-language-action models struggle with partial observability and task-progress retention during long-horizon manipulation with only a wrist camera. AtlasVLA uses persistent world and ego memories to enable proactive reasoning, achieving state-of-the-art performance across simulation and real-world benchmarks while improving LIBERO-Long success by 9.4%.

  • Problem

    Reactive VLAs lack persistent spatial and temporal state, limiting wrist-only manipulation in partially observable, long-horizon tasks.

  • Method

    AtlasVLA combines voxel-hashed Persistent World Memory and Ego-Working Memory, conditioning a diffusion transformer on their joint state for wrist-only action generation.

  • Results

    AtlasVLA achieves state-of-the-art performance across simulation and real-world benchmarks using solely a wrist camera, improving LIBERO-Long success by +9.4%.

  • Takeaways & Limitations

    Persistent world-ego state supports robust wrist-only manipulation under partial observability and extended-horizon demands.

Abstract

from arXiv · show

While Vision-Language-Action (VLA) models have advanced embodied AI, their fundamentally reactive paradigm severely limits performance in partially observable and long-horizon tasks. When restricted to a single wrist-mounted camera, they inevitably suffer from perception forgetting as objects exit the field of view, and temporal task-progress forgetting} during multi-step execution. To overcome these bottlenecks, we propose AtlasVLA, a novel framework that transitions from direct reactive manipulation to proactive reasoning through a persistent world-ego state. AtlasVLA features a dual-memory architecture: a 4D Persistent World State Memory that lifts transient 2D observations into a globally updated, voxel-hashed spatial state to resolve visual blind spots, and an Ego-Working State Memory that tracks historical ego state and task progress. By conditioning a diffusion transformer (DiT) on this joint World-Ego state, AtlasVLA enables robust spatial reasoning. Extensive evaluations across LIBERO, RLBench, and real-world benchmarks demonstrate that AtlasVLA achieves state-of-the-art performance using solely a wrist camera. Remarkably, it decisively outperforms multi-view baselines, yielding absolute success rate improvements of 9.4% on LIBERO-Long and 17.5% in real-world long-horizon tasks.

1 Introduction

AtlasVLA addresses spatial partial observation and temporal task-progress forgetting in wrist-only reactive VLAs by maintaining a persistent world-ego state. Its dual-memory architecture supports proactive long-horizon manipulation and achieves state-of-the-art performance across simulated and real-world tasks.

  • Limitations: Reactive wrist-only VLAs forget task-relevant objects and spatial structure when the dynamically shifting camera field of view no longer observes them.Instantaneous camera observations do not equal the true world state, causing agents to lose track of their surroundings.
  • Limitations: Reactive models also suffer task-progress forgetting because they lack historical context for completed subtasks, ego state, and remaining actions.This limitation is especially damaging in long-horizon manipulation requiring multiple sequential sub-steps.
  • AtlasVLA: AtlasVLA introduces a dual-memory framework combining a 4D Persistent World Memory for spatial blind spots with an Ego-Working Memory for task progress.The framework continuously reconstructs and updates hidden environmental state from limited egocentric observations, shifting from reactive observation-action behavior toward proactive reasoning.
  • AtlasVLA: The Persistent World State Memory lifts transient 2D wrist observations into a globally updated 4D voxel-hashed state, while the Ego-Working State Memory tracks historical ego state and task progress.The world memory uses depth estimation, spatial back-projection, neighborhood fusion, and sliding windows; the combined state conditions a step-wise diffusion transformer.
  • Results: 97.6% success rate on LIBERO and 70.8% on RLBench establish AtlasVLA’s state-of-the-art performance using solely a wrist-mounted camera.The paper states that AtlasVLA decisively outperforms representative multi-view baselines.

2 Related Work

Recent VLA models build on LLMs and VLMs to advance embodied AI, including approaches that use iterative denoising for complex multimodal continuous trajectories. However, many methods still depend on multi-view or third-person observations, leaving the wrist-only bottleneck unresolved.

  • Vision-Language-Action Models: Recent VLA models build on LLMs and VLMs, continuing a paradigm shift toward embodied artificial intelligence.The passage situates VLA development within advances in large language and vision-language models.
  • Vision-Language-Action Models: Iterative denoising enables recent approaches to synthesize complex, multimodal continuous trajectories.This describes a generative modeling strategy used by the cited methods.
  • Vision-Language-Action Models: Many existing methods depend on multi-view observation configurations or third-person setups, failing to resolve the wrist-only bottleneck.The passage identifies both observation requirements as a limitation of prior approaches.

3 Method

AtlasVLA formulates wrist-only VLA manipulation as sequential action-chunk prediction conditioned on visual, proprioceptive, and language inputs. Its dual persistent world-ego memory lifts observations into a globally updated 4D representation while tracking intent and task progress for action generation.

  • Problem Formulation: AtlasVLA restricts the agent to wrist-mounted visual observations, proprioceptive state, and a language instruction when predicting future action chunks.Each action is a 7-dimensional vector containing the 6-DoF end-effector pose and binary gripper state.
  • Persistent World State Memory: The Persistent World State Memory lifts 2D visual tokens into 3D using depth and spatial back-projection, then adds spatial and temporal embeddings to form a 4D representation.Depth Anything v3 supplies depth, while MLP-parameterized embeddings encode 3D position and temporal order.
  • World-Ego-Guided Action Generation: Historical ego context retrieves relevant world states from the global memory, and the resulting world-ego context conditions the action diffusion transformer for precise action generation.Ego-guided cross-attention suppresses spatial redundancy while extracting task-relevant world states.
  • Persistent World State Memory: The global world memory updates through confidence-weighted voxel-hashing fusion and a temporal sliding window, while permanently anchoring the first-frame spatiotemporal memory.Higher-confidence depth observations contribute more to aggregation, and older states are forgotten within the maximum window W.
  • Ego-Working Memory: Intent-aware queries condense spatiotemporal context into ego-working tokens that track task intent, state, and progress, explicitly addressing intention forgetting in multi-stage tasks.The learnable intent queries Qego aggregate goal-oriented information from the VLM, while consolidation merges adjacent and semantically similar intent tokens.

4 Experiments

Experiments evaluate AtlasVLA under strict wrist-only settings across LIBERO, RLBench, and real-world manipulation suites, with ablations isolating the contributions of its memory modules. Results show that persistent world and ego-working memory are critical for robust spatial reasoning and long-horizon task execution.

  • Benchmark Setup: AtlasVLA is evaluated on LIBERO’s five suites, RLBench, and real-world general and long-horizon task suites under wrist-only camera constraints.The evaluations cover simulation and real-world manipulation, including sequential, rearrangement, and multi-stage tasks.
  • Simulation Results: 3.4% higher than π0, AtlasVLA outperforms the π0 model equipped with additional third-person views by accumulating spatio-temporal context from sequential local observations.This comparison highlights the benefit of persistent context over instantaneous third-person observations.
  • RLBench Evaluation: The RLBench evaluation uses a single 128 × 128 RGB wrist view, discards all third-person views, and averages success across six representative tasks over 20 trials each.Models are trained for 80k steps using 100 demonstrations per task.
  • Ablation Studies: 69.5% →54.0%, removing world state memory causes a catastrophic real-world performance collapse, demonstrating the indispensability of spatial map memory under wrist-only control.The ablation study evaluates architectural components on LIBERO and real-world long-horizon tasks.
  • Ablation Studies: 13.0% drop, removing ego-working memory harms performance by causing intention drift on long-horizon tasks.The result supports the role of latent ego-working memory in retaining task progress.

5 Conclusion

AtlasVLA addresses reactive VLA models’ lack of persistent world state for robust wrist-only manipulation under partial observability. Its dual-memory design preserves global world state and tracks task progression to prevent intention drift.

  • 5 Conclusion: AtlasVLA addresses the lack of persistent world state in reactive VLA models, enabling robust wrist-only manipulation under severe partial observability.It introduces a Persistent World Memory with a voxel-hashed spatial map to preserve global world state.
  • 5 Conclusion: Its Ego-Working Memory tracks task progression and prevents intention drift during manipulation.The framework combines this memory with Persistent World Memory to support action generation conditioned on world state and task progress.

A Implementation Details · A.1 AtlasVLA: Architecture

AtlasVLA’s architecture combines persistent world-ego state memory with three interconnected modules: egocentric visual encoding, semantic reasoning, and diffusion-transformer action generation. Its visual encoder builds multimodal inputs from wrist-camera RGB and streaming depth, while the language model tracks intent and the action expert produces continuous robotic control.

  • A.1 AtlasVLA: Architecture: AtlasVLA is organized around persistent world-ego state memory and three interconnected modules for perception, reasoning, and action generation.The modules are a multimodal visual encoder, a large language model, and a step-wise conditioned Diffusion Transformer.
  • A.1 AtlasVLA: Architecture: The dual-stream visual encoder extracts 2D visual tokens from wrist-camera RGB observations and estimates streaming depth for persistent 4D world representation.The RGB stream uses frozen DINOv2 and SigLIP encoders, while the spatial stream uses fine-tuned Depth Anything v3.
  • A.1 AtlasVLA: Architecture: Frozen DINOv2 and SigLIP encoders process the primary RGB stream to extract visual tokens directly from egocentric wrist-camera observations.These encoders support the RGB component of AtlasVLA’s dual-stream perception module.
  • A.1 AtlasVLA: Architecture: Fine-tuned Depth Anything v3 supplies robust streaming depth estimation for constructing AtlasVLA’s persistent 4D world representation.This depth estimation is provided by the spatial stream of the visual encoder.
  • A.1 AtlasVLA: Architecture: A LLaMA-2 7B decoder-only language model processes text instructions, visual tokens, world state tokens, and learnable intent queries for semantic reasoning.AtlasVLA uses OpenVLA-7b as its pre-training model, and the LLM serves as a high-level intent tracker rather than directly decoding low-level actions.
  • A.1 AtlasVLA: Architecture: An action expert with ∼300M weights uses an augmented Diffusion Transformer to translate abstract world-ego states into continuous robotic control.The DiT architecture is inspired by CogACT and is adapted to support AtlasVLA’s dual-memory formulation.

A.2 Additional Training Details

AtlasVLA trains a step-wise conditioned DiT to reconstruct ground-truth action chunks from Gaussian noise using dual-state conditioning. Training additionally supports classifier-free guidance through independent condition dropout.

  • Loss Function: The action expert reconstructs ground-truth action chunks from Gaussian noise while being strictly conditioned on dual-state representations.The continuous action synthesis is modeled as a denoising process parameterized by a step-wise conditioned Diffusion Transformer.
  • Loss Function: The forward diffusion process adds Gaussian noise ϵ ∼N(0, I) to each ground-truth action chunk At over K steps.The noise schedule uses a predefined cosine schedule through ¯αk.
  • Loss Function: During reverse diffusion, DiT predicts injected noise using sequentially decoupled conditioning from temporal ego progress and geometrically grounded world state.The denoising objective is formulated as Mean Squared Error loss.
  • Loss Function: Conditioning contexts are independently replaced with a learnable unconditional null token at pdrop = 0.1 to enable classifier-free guidance during inference.This trains the model both conditionally and unconditionally for guided sampling.
  • Hyperparameter Setting: Training uses a global batch size of 256 across 8 GPUs and a learning rate of 2 × 10−5, while inference denoising runs for 10 steps with action chunks of 16.The architecture uses 2048 persistent-world tokens, 4 ego-working tokens, 2 ego retrieval cross-attention layers, and 4 world retrieval layers.

B Additional Experimental Results · B.1 Additional Simulation Details · B.2 Additional Real-world Settings

The additional experiments detail simulation coverage and real-world evaluation settings for testing compositional generalization, sequential decision-making, and wrist-only spatial reasoning. They specify benchmark suites, hardware constraints, randomized trials, and success criteria.

  • B.1 Additional Simulation Details: LIBERO evaluates embodied agents across five task suites: LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, LIBERO-Long, and LIBERO-90.These suites assess compositional generalization and sequential decision-making under diverse initial conditions.
  • B.1 Additional Simulation Details: LIBERO tasks require agents to accomplish coherent series of sub-goals under diverse initial conditions.This structure poses a severe challenge for standard memoryless agents.
  • B.2 Additional Real-world Settings: Real-world evaluations use a 7-DoF Franka robotic manipulator.The hardware setup tests the proposed system on a physical robotic platform.
  • B.2 Additional Real-world Settings: The system uses only one Intel RealSense D415 camera mounted on the end-effector for egocentric perception.Third-person global cameras are disabled and disconnected to test spatial reasoning under severe partial observability.
  • B.2 Additional Real-world Settings: Each real-world task is evaluated across 50 independent execution trials with randomized object placements and varied spatial layouts.The trials use identical robot configurations.
  • B.2 Additional Real-world Settings: A rollout counts as successful only when the agent sequentially completes all required sub-goals and reaches the specified final target state within a designated step limit.This criterion is applied during real-world inference evaluation, including the demanding long-horizon suite.

B.3 Runtime Analysis

AtlasVLA’s dual-memory architecture is evaluated for computational efficiency against MemoryVLA and maintains highly competitive inference efficiency despite integrating persistent world-state updates and ego-state retrieval.

  • Runtime Analysis: AtlasVLA is evaluated against MemoryVLA as a strong baseline for real-world robotic deployment runtime efficiency.The comparison focuses on the computational efficiency of the proposed dual-memory architecture.
  • Runtime Analysis: Despite continuous 4D Persistent World State updating and step-wise Ego-Working State retrieval, AtlasVLA maintains highly competitive inference efficiency.These mechanisms are integrated into the dual-memory architecture evaluated in the runtime analysis.

B.4 Detailed Ablation Study

The ablations identify 16 as the optimal Ego-Working State memory length and 0.025m as the best Persistent World State voxel size. Longer or finer-grained configurations provide no consistent benefit and can degrade performance.

  • Ego-Working State Memory Length: Expanding memory to 32 yields 69.8% on real-world tasks but lowers LIBERO performance to 97.2%, indicating redundant historical noise.The added history dilutes cross-attention without meaningful gains, so memory length 16 is adopted.
  • Persistent World State Voxel Size: A voxel size of 0.025m achieves the highest success rates on LIBERO and real-world long-horizon tasks, with 97.6% and 69.5%, respectively.Reducing voxel size to 0.01m degrades performance to 96.3% on LIBERO and 65.7% on real-world tasks, likely due to spatial sparsity and representation noise.

B.5 Qualitative Results

AtlasVLA’s qualitative trajectories demonstrate robust wrist-only manipulation under strict partial observability in real-world and simulated environments. The visualizations highlight its ability to mitigate perception forgetting and intention drift during complex, multi-stage behaviors.

  • Real-world qualitative results: AtlasVLA executes step-by-step trajectories for real-world “Pepper in Box” and “Can in Drawer” pick-and-place and spatial rearrangement tasks.These tasks are performed under strict wrist-only partial observability, where the narrow field of view causes the target receptacle to leave view.
  • Persistent state modeling: The Persistent World State mitigates both intention drift and perception forgetting during complex, multi-stage behaviors.The passage presents this as the mechanism underlying robust execution under partial observability.
  • LIBERO qualitative results: AtlasVLA’s LIBERO trajectories further visualize robust manipulation capabilities enabled by its dual-memory architecture.Figure 7 presents step-by-step executions across the comprehensive LIBERO benchmark suites.
Loading 2608.06729v1…