AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agents

Pan Wang; Yihao Hu; Xiujin Liu; Jingchu Yang; Hang Wang; Zhihao Wen

Source-linked AI summary

AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agents

Pan Wang, Yihao Hu, Xiujin Liu, Jingchu Yang, Hang Wang, Zhihao Wen

arXiv:2605.17933v1May 18, 2026cs.CV

TL;DR

VLM agents often store spatial experience as lossy text and rely on external teacher models, limiting visually grounded reuse. AtlasVA introduces teacher-free Visual Skill Memory with visual atlases and dense reward shaping, achieving 0.93 average success across benchmarks and outperforming larger models. Its 2.5D priors remain limited for highly occluded, ego-centric 3D robotics.

Problem
Text-centric memory compresses spatial structure into lossy language and often depends on external teacher models, poorly matching visually grounded VLM decision making.
Method
AtlasVA combines spatial heatmaps, visual exemplars, and symbolic skills with teacher-free atlas evolution from trajectories and potential-based visual reward shaping.
Results
0.93 average success rate across evaluated benchmarks, outperforming GPT-5 (0.69), o3 (0.71), and VAGEN (0.78).
Takeaways & Limitations
Visually grounded memory and dense reward shaping can improve VLM performance across spatially intensive 2D and 3D tasks without external teacher supervision.
Takeaways & Limitations
AtlasVA projects tabletop 3D manipulation into 2.5D visual priors, limiting demonstrated scope for highly occluded, ego-centric 3D robotics.

Abstract

from arXiv · show

Vision-language model (VLM) agents increasingly rely on memory-augmented reinforcement learning to reuse experience across long-horizon tasks, yet most existing frameworks store memory as text and depend on proprietary teacher models to summarize or refine it. This design is poorly matched to spatial decision making: geometric priors are compressed into lossy language, and sparse interaction is often supervised through delayed textual feedback rather than dense visually grounded signals. We argue that reusable experience for VLM agents should remain visually grounded. Based on this insight, we propose \textbf{AtlasVA}, a teacher-free visual skill memory framework that organizes memory into three complementary layers: spatial heatmaps, visual exemplars, and symbolic text skills. AtlasVA further evolves danger and affinity atlases directly from trajectory statistics and lightweight grid heuristics, and reuses these self-evolving atlases as potential-based shaping rewards for reinforcement learning. This unifies perception, memory, and optimization without external LLM supervision. Experiments on \textsc{Sokoban}, \textsc{FrozenLake}, 3D embodied navigation, and 3D robotic manipulation benchmarks show that AtlasVA consistently outperforms text-centric memory baselines and competitive VLM agents, with especially strong gains on spatially intensive tasks. Homepage: https://wangpan-ustc.github.io/AtlasvaWeb

1 Introduction

Long-horizon spatial tasks require VLM agents to accumulate reusable experience, but text-centric memory loses visual structure, depends on proprietary teacher models, and mismatches spatial feedback. AtlasVA addresses these limitations with a teacher-free Visual Skill Memory hierarchy combining spatial heatmaps, visual exemplars, and symbolic text.

Motivation: VLM agents must parse instructions, interpret screenshots, and execute grounded actions while accumulating reusable experience in long-horizon spatial tasks.This setting is challenging because agents cannot simply reason from scratch at every step.
Limitations of Existing Memory: Existing memory frameworks summarize trajectories into text, retrieve relevant summaries, and use strong teacher models to refine skills over time.SkillRL and XSkill exemplify this recipe for exploration and experience reuse.
Limitations of Existing Memory: Text-centric memory loses rich spatial details, requires proprietary LLM supervision, and creates feedback mismatch when evaluating spatial interactions.Text struggles with topologies such as dead ends, local hazards, and promising sub-goal regions; proprietary models also increase computational costs and undermine autonomous self-improvement.
AtlasVA: AtlasVA replaces text-only storage with a Visual Skill Memory hierarchy of spatial heatmaps, visual exemplars, and symbolic text.During inference, heatmaps act as visual prompts that project complex 3D environments into 2.5D spatial maps while preserving spatial knowledge within the VLM’s native modality.
AtlasVA: AtlasVA evolves danger and affinity heatmaps directly from raw trajectory statistics through a teacher-free mechanism, removing proprietary LLM supervision.The framework aligns reusable experience with visual perception while sustaining its memory without external teacher models.

2 Related Work

Related work establishes VLM agents as capable sequential decision-makers, but highlights a reliance on transient visual inputs, text-compressed experience, and externally generated textual rewards. Recent RL methods add memory and skill evolution, while potential-based reward shaping offers a formal response to sparse terminal rewards.

VLM Agents: VLM agents combine visual grounding with instruction following for sequential decision-making in interactive environments, including web navigation, GUI control, and embodied planning.The supplied passages describe these capabilities across several interactive domains.
Memory-Augmented Agents: Most architectures treat visual observations as transient inputs and compress historical context or long-horizon heuristics into verbal summaries, scratchpads, or retrieved passages.This creates an architectural asymmetry between visual decision-making and text-based memory.
Memory and Skill Evolution: Recent RL frameworks support self-improvement through external memory and skill evolution, using verbal self-critique or retrievable linguistic records and natural-language skill libraries.Examples include Reflexion, ExpeL, Mem0, SkillRL, and XSkill.
Reward Shaping: Sparse terminal rewards cause severe sample inefficiency in spatial reasoning, motivating textual rewards generated by external LLMs from action logs.Potential-based reward shaping formally addresses this by adding the difference of a potential function Φ to the reward.

3 AtlasVA

AtlasVA equips VLM agents with a teacher-free, multimodal Visual Skill Memory combining spatial heatmaps, visual exemplars, and symbolic text skills. It self-evolves spatial priors from rollout data and uses them for dense reward shaping during reinforcement learning.

Problem Formulation: AtlasVA formulates embodied decision-making as a partially observable Markov decision process with multimodal observations and discrete actions sampled from a VLM policy.
Visual Skill Memory: The Visual Skill Memory prompts the VLM with three complementary layers: spatial heatmaps, visual exemplars, and symbolic text skills.The layers provide perceptual spatial maps, annotated screenshots, and compact textual heuristics.
Visual Skill Memory: Spatial heatmaps render danger and affinity fields as RGB images aligned with the VLM visual encoder, rather than lossy text arrays.Danger maps indicate deadlock risks, while affinity maps indicate proximity to task completion.
Teacher-Free Self-Evolution: AtlasVA bootstraps spatial priors without external LLM teachers by fusing static grid heuristics with accumulated trajectory statistics.Statistics use failed-trajectory terminal positions for danger and successful-path coordinate frequencies for affinity, then update historical maps through EMA blending.
Teacher-Free Self-Evolution: The scheduling coefficient β_k anneals from 0 to 1, transitioning from static-geometry guidance during cold-start exploration to experience-driven refinement later in training.This transition is designed to avoid catastrophic forgetting.
Reward Shaping: AtlasVA uses the self-evolving spatial atlas as a dynamic potential function, combining danger penalties and affinity gains into a bounded auxiliary visual reward.The final reinforcement-learning reward is r_visual + r_env; affinity uses potential differences, while danger penalties prioritize safe navigation over hazardous shortcuts.

4 Experiments

Across 2D and 3D benchmarks, AtlasVA achieves strong spatial-task performance with a compact 3B-parameter VLM, while visual memory and atlas-based shaping improve convergence and grounding. Ablations and trajectory-only heatmap evolution support the contribution of its visual skill components.

Efficacy in Spatial Reasoning: AtlasVA raises Sokoban success from zero-shot Qwen2.5-VL-3B’s 0.14 to 0.79, exceeding GPT-5’s 0.70.The gains are reported for environments requiring intensive geometric planning.
Robustness in 3D Navigation and 3D Robotic Manipulation: AtlasVA reaches 0.86 average success in 3D Navigation and 1.00 across Place, Stack, Drawer, Align, and Swap manipulation tasks.The manipulation results use PrimitiveSkill via ManiSkill.
Learning Efficiency and Convergence: AtlasVA climbs to approximately 0.80 Sokoban success within 140 training steps, whereas the text-only Layer 3 baseline struggles to exceed 0.25.The comparison attributes the baseline difficulty to modality mismatch and sparse feedback.
Evolution of Spatial Heatmaps: By Step 200, EMA-updated heatmaps capture meaningful topological structures, encoding hazards and sub-goal paths through pure environment interaction.The progression begins with spatially uninformative heatmaps at Step 0 and supports teacher-free visual-prior extraction.
Dynamics of the Visual Exemplar Pool: The exemplar pool reaches its six-exemplar capacity within 40 training steps, coinciding with validation success rising from near zero to over 70%.Active exemplar retrieval keeps visual context aligned with the current exploration frontier while bounding prompt size.

5 Conclusion and Limitations … B Prompt Template

AtlasVA combines teacher-free visual skill memory with trajectory-derived spatial priors and dense reward shaping, while formalizing these components as an augmented decision process. The paper concludes with strong spatial-task performance, a 2.5D manipulation limitation, and a fixed-anchor prompt template for its three memory layers.

5 Conclusion and Limitations: AtlasVA resolves spatial blindness and modality mismatch by bootstrapping geometric priors from trajectories into continuous visual reward shaping without external teacher supervision.Its native Visual Skill Memory retains reusable spatial information in a visually grounded form.
5 Conclusion and Limitations: A compact 3B-parameter AtlasVA significantly outperforms larger models such as GPT-5 across diverse spatial benchmarks.The conclusion identifies this result as evidence of AtlasVA’s effectiveness despite its smaller parameter count.
5 Conclusion and Limitations: The primary limitation is projecting table-top 3D manipulation into 2.5D visual priors, leaving highly occluded, ego-centric 3D robotics as critical future work.The limitation concerns extending the current spatial representation beyond table-top settings.
Appendix: The appendix formalizes AtlasVA’s Visual Skill Memory and dense reward shaping as modifications to a POMDP with an evolving observation space and dynamic potential-based reward.This formal treatment explains how the components alter the underlying decision-making process.
A.1 Standard POMDP vs. Augmented Observation Space: AtlasVA augments the standard POMDP observation with its three-layer Visual Skill Memory, making non-local spatial priors native to the state representation instead of relying on textual coordinate inference.The memory state is denoted Mk, and the augmented observation is denoted ˜ot.
A.2 Formalization of Potential-Based Dense Reward: AtlasVA converts sparse environment rewards into dense per-step rewards by combining affinity potential differences with a non-potential-based danger penalty.The affinity branch uses a BFS distance field and trajectory statistics to provide gradients at every state, while the danger branch penalizes deadlock or fatal-trap regions.
A.3 Off-Policy Memory Evolution Dynamics: AtlasVA evolves spatial priors off-policy through EMA updates that aggregate terminal failures into danger maps and successful-path visitation frequencies into affinity maps.These statistics are blended with BFS, corner, wall-adjacency, and hole-neighborhood heuristics.
A.4 Algorithmic Formalization of AtlasVA Evolution: Algorithm 1 formalizes teacher-free visual atlas evolution and atlas-grounded visual reward shaping by extracting visual priors from raw trajectories and converting them into dynamic potential functions.The algorithm focuses on AtlasVA’s core contributions rather than standard multi-turn PPO loops.

C Implementation and Environmental Details … C.4 Environment Details and Reward Assignment

Appendix C specifies AtlasVA’s three-layer visual skill memory, its teacher-free evolution and retrieval procedures, and its dense reward-shaping components across evaluated environments. The implementation preserves spatial statistics as visual references while grounding symbolic rules in environment specifications.

C Implementation and Environmental Details: Appendix C details Visual Skill Memory construction and the reward-assignment hyperparameters used across evaluated environments.The section covers implementation and environmental details for the evaluated settings.
C.1 Layer 1: Spatial Heatmaps (Dense Perceptual Priors): AtlasVA stores danger and affinity statistics on a discrete 2D grid, renders them as pure red and green alpha-channel heatmaps, and supplies them as separate auxiliary image tokens.Danger uses R=255, G=0, B=0; affinity uses R=0, G=255, B=0, with opacity proportional to tensor values. Separate tokens avoid corrupting recognition of small foreground objects.
C.1 Layer 1: Spatial Heatmaps (Dense Perceptual Priors): During PPO, AtlasVA combines sparse environment rewards with affinity progress and danger penalties, then updates the policy using the resulting dense visual rewards.The shaping terms are defined as raffinity = Maffinity(p_t+1) − Maffinity(p_t) and rdanger = −β · Mdanger(p_t+1).
C.1 Layer 1: Spatial Heatmaps (Dense Perceptual Priors): AtlasVA evolves danger maps from failed trajectories, affinity maps from successful trajectories, and exemplar memory from critical keyframes using EMA updates and DINOv2-based FIFO retrieval.The update procedure maintains Mdanger, Maffinity, and Evis as the returned Visual Skill Memory.
C.2 Layer 2: Visual Exemplars (Episodic Context): Layer 2 retrieves the top-k structurally relevant visual exemplars using cosine similarity between frozen DINOv2 embeddings, then separates them into positive and negative cases for zero-shot visual matching.The pool contains inflection-point frames such as frames before irreversible deadlocks or after critical sub-goal completion; DINOv2 is selected for patch-level spatial correspondences.
C.3 Layer 3: Symbolic Text Skills (Semantic Grounding): Layer 3 constructs symbolic text skills directly from environment rulebooks and task descriptions, without proprietary teacher-LLM generation or summarization.These rules provide foundational logic, while heatmaps and retrieved exemplars ground that logic in the current visual state.
C.4 Environment Details and Reward Assignment: AtlasVA adds λdanger penalties for entering danger-map deadlock regions and λaffinity gains for moving toward affinity-map goal regions, while retaining sparse binary base rewards.The framework also includes a format penalty for malformed actions or invalid JSON.
C.4 Environment Details and Reward Assignment: For complex 3D tasks, visual shaping adds rotation alignment for 3D Navigation and stage-success deltas for multi-step PrimitiveSkill manipulation while preserving sparse base-environment rewards.These task-specific heuristics extend the proposed dense shaping mechanism.

C.5 GridState Abstraction and Simulator APIs

GridState extracts exact simulator coordinates to build spatial heatmaps and dense visual rewards, with Table 4 documenting the accessed APIs. This privileged state information is restricted to training; evaluation uses only RGB observations and rendered visual memory prompts.

GridState abstraction: GridState extracts exact 2D/3D coordinates from each simulator to construct spatial heatmaps and compute dense visual rewards.Table 4 specifies the low-level simulator APIs accessed for each environment.
Training-time access: Privileged simulator states are used exclusively during training for offline atlas evolution and reward computation, not as policy inputs.This restriction preserves the vision-based nature of evaluation comparisons.
Evaluation-time inputs: At evaluation, the VLM policy receives only raw RGB observations and rendered heatmap and exemplar prompts.The policy does not receive low-level simulator states.

C.6 Optimization and Hyperparameter Configurations

The paper uses unified PPO and Visual Skill Memory configurations across experiments, with a fixed VLM, asynchronous rollout engine, and consistent teacher-free memory settings. These configurations support multimodal memory within strict token limits and test robustness across environments.

Optimization and Hyperparameter Configurations: Experiments use unified PPO and VSM hyperparameters on 8 × NVIDIA RTX 6000 Ada Generation GPUs with Qwen2.5-VL-3B-Instruct and vLLM asynchronous rollouts.Prompt and response token lengths are strictly capped to accommodate injected multimodal memory.
Optimization and Hyperparameter Configurations: VSM settings remain fixed across environments, using EMA decay λ = 0.85 and a visual exemplar pool containing 3 positive and 3 negative cases.The consistent settings are used to demonstrate the robustness and generalizability of the teacher-free evolution mechanism.

D Additional Qualitative Analysis · D.1 Projection of Continuous 3D Spaces into 2.5D Visual Priors

AtlasVA projects continuous 3D navigation and manipulation spaces into 2.5D visual priors through the GridState module. During reward shaping, inverse coordinate mapping retrieves evolving spatial potentials that provide dense, interpretable guidance for 3D control.

D.1 Projection of Continuous 3D Spaces into 2.5D Visual Priors: AtlasVA addresses continuous 3D navigation and robotic manipulation by introducing a dimensionality-reduction abstraction through the GridState module.This abstraction connects continuous physical spaces with AtlasVA’s visual atlas representation.
D.1 Projection of Continuous 3D Spaces into 2.5D Visual Priors: For 3D navigation, AtlasVA projects continuous reachable surfaces onto the X-Z plane and dynamically discretizes rooms into 2D floor plans.The stated discretization resolution is 0.25 meters per cell.
D.1 Projection of Continuous 3D Spaces into 2.5D Visual Priors: The 3D navigation projection uses a resolution of 0.25 meters per cell.This resolution defines the spatial granularity of the dynamically constructed floor plan.
D.1 Projection of Continuous 3D Spaces into 2.5D Visual Priors: For 3D robotic manipulation, AtlasVA maps the continuous operating table to a localized 2.5D workspace grid.The passage describes this representation under the PrimitiveSkill setting.
D.1 Projection of Continuous 3D Spaces into 2.5D Visual Priors: During reward shaping, AtlasVA inversely maps the agent’s real-time continuous 3D coordinates onto the 2.5D heatmap.This inverse mapping connects physical coordinates with the visual prior used for shaping.
D.1 Projection of Continuous 3D Spaces into 2.5D Visual Priors: The mapped coordinates let AtlasVA retrieve evolving spatial potentials and generate dense, coordinate-aligned gradients.These gradients guide agent behavior using spatial information aligned with the agent’s current position.
D.1 Projection of Continuous 3D Spaces into 2.5D Visual Priors: AtlasVA therefore guides complex 3D manipulations with interpretable 2.5D visual priors.The passage associates this guidance with significantly reduced sample complexity in 3D continuous reinforcement learning.

D.2 Visualization of Atlas-Grounded Visual Reward Shaping

The section visualizes step-by-step dense reward dynamics in a successful Sokoban rollout, showing how visual potential fields address credit assignment under sparse rewards. Without intermediate feedback, VLM agents typically take random walks or enter deadlocks.

Visualization of Atlas-Grounded Visual Reward Shaping: Figure 11 visualizes the step-by-step dense reward dynamics of a successful Sokoban rollout.The visualization is intended to provide a concrete understanding of how visual potential fields alleviate credit assignment in sparse-reward environments.
Visualization of Atlas-Grounded Visual Reward Shaping: Sokoban provides only a sparse extrinsic reward of +1 upon completing the entire task.No intermediate feedback is provided during the rollout.
Visualization of Atlas-Grounded Visual Reward Shaping: Without intermediate feedback, VLM agents typically resort to random walks or fall into deadlocks.The section motivates dense visual reward shaping as a response to these sparse-reward failure modes.

PPO & GAE

AtlasVA replaces sparse terminal feedback with dense, spatially grounded visual reward gradients for reinforcement learning. Danger penalties provide immediate corrective feedback near deadlocks, while affinity gains sustain progress along viable sub-goal paths.

Visual reward shaping: AtlasVA provides a continuous, coordinate-specific reward gradient throughout each episode.The mechanism transforms a single terminal sparse reward into a continuous optimization landscape over time.
Danger penalties: Danger penalties immediately penalize actions moving toward irreversible corners or wall traps, prompting corrective backtracking before terminal failure.The visual potential field Φdanger supplies negative feedback when the agent approaches deadlocks.
Affinity gains: Affinity gains provide sustained positive rewards as the agent pushes boxes along structurally valid paths toward the goal manifold.The Φaffinity potential supplies continuous step-level guidance for long-horizon tasks.
Optimization impact: Replacing sparse feedback with dense, spatially grounded visual gradients stabilizes AtlasVA’s multi-turn reinforcement-learning optimization.The shaping mechanism bridges the modality gap between visual spatial decisions and optimization signals.

D.3 Qualitative Rollout Comparison

Figure 12 compares AtlasVA with Qwen2.5-VL-72B across Sokoban, FrozenLake, and PrimitiveSkill Swap, showing contrasting spatial reasoning trajectories. The baseline exhibits spatial blindness, whereas AtlasVA uses self-evolved visual heatmaps to ground geometric planning, even from a compact 3B model.

Comparison Scope: Figure 12 compares AtlasVA against Qwen2.5-VL-72B across Sokoban, FrozenLake, and PrimitiveSkill Swap rollout trajectories.The comparison targets three spatially demanding environments.
AtlasVA Trajectories: AtlasVA injects self-evolved Danger and Affinity heatmaps, visually marking deadlocks and optimal sub-goal paths to transform geometric planning into visual pattern matching.Red regions denote Danger, while green regions denote Affinity.
AtlasVA Trajectories: AtlasVA successfully navigates these spatial bottlenecks even when initialized from a compact 3B parameter model without external text summarization.The passage attributes this capability to multimodal spatial grounding through self-evolved spatial heatmaps.