PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoMem-VLA: Internalizing 3D Scene Understanding into Temporal Memory for Vision-Language-Action Models

Qinfan Zhang, Yifan Luo, Haoyuan Shi, Guanlin Liu, Han Dong, Haozhe Shan, Yong Dai, Xiangcong Ren, Xiaozhu Ju

arXiv:2610.29866v1cs.CV

TL;DR

Current-view 2D VLA representations offer limited support for precise 3D reasoning that depends on interaction history. GeoMem-VLA trains geometry-supervised tokens and retains them in causal episode memory; it reports 98.9% average success on LIBERO and 91.10% / 91.24% on RoboTwin Clean / Randomized.

  • Problem

    Current-observation-based VLA models offer limited support for precise 3D spatial reasoning and decision-making that depend on interaction history.

    Source: §1 · p. 1

  • Method

    GeoMem-VLA trains GeoPlan tokens on current-scene point maps and future end-effector tracks, then couples geometric representations with causal memory for action generation.

    Source: §1 · p. 2, §3.2 · p. 5, §3.3 · p. 5

  • Results

    GeoMem-VLA reports 98.9% average success across the four LIBERO suites and 91.10% / 91.24% on RoboTwin Clean / Randomized.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Results across manipulation benchmarks and physical robot experiments support geometry-supervised latent memory as a practical connection between 3D representation learning and action generation over time.

    Source: §5 · p. 9

    Evaluation covers object manipulation in simulation and on physical robots, not human–robot interaction or unconstrained settings; reported task success does not establish safety around people.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Geometric structure is essential for robotic manipulation and understanding the 3D world. Most VLA models rely on 2D representations of current observations, while geometry-aware approaches often require additional 3D information. However, geometry inferred independently from individual observations is often discarded rather than accumulated into a persistent task representation. Our key insight is that action generation needs a compact, task-relevant geometric abstraction that can be continuously updated, rather than a complete, renderable 3D reconstruction at every step. We introduce GeoMem-VLA, a VLA framework that internalizes explicit 3D supervision into dedicated geometric tokens and maintains them as temporal memory for action generation, without directly aligning the VLM's native visual embeddings with the teacher feature space. During training, a frozen geometry foundation model supervises learnable GeoPlan tokens through a geometric flow-matching objective over the current scene point map and future end-effector tracks. These tokens are integrated with the memory of observation representations, where historical information is retrieved, adaptively fused with current observations, and consolidated to retain event frames as the task progresses. At inference, the policy retrieves historical representations and fuses them with the current working representation to predict actions, without invoking the 3D teacher or explicitly reconstructing the scene. After benchmark-specific downstream adaptation, GeoMem-VLA achieves 98.9% average success on LIBERO and 91.10% / 91.24% on RoboTwin's clean/randomized settings. Across five real-world tasks, GeoMem-VLA achieves 78.0% average success. These results suggest that internalizing and remembering 3D scene understanding is a promising approach to robot control.

1 INTRODUCTION

GeoMem-VLA addresses the limited support current 2D observations provide for precise 3D reasoning that depends on interaction history. It learns geometry-supervised GeoPlan representations and retains them in causal memory for action generation.

1 INTRODUCTION

  • Current-observation-based VLA models offer limited support for precise 3D spatial reasoning and decisions that depend on interaction history.The introduction frames the central question as how geometric supervision can shape spatial memory that guides action.Source: §1 · pp. 1–2
  • GeoMem-VLA trains learnable GeoPlan tokens to predict current-scene point maps and future Cartesian end-effector tracks, then uses them as compact geometric context at deployment.A frozen VGGT teacher provides scene-geometry targets, while demonstration trajectories provide future-motion supervision; the teacher and reconstruction head are removed from the action-generation path.Source: §1 · p. 2
  • Causal retrieval, adaptive fusion, and similarity-based consolidation maintain geometric, perceptual, and task context while retaining feature-distinct temporal states.Event-aware training samples temporally separated observations from the same episode in chronological order.Source: §1 · p. 2
  • 98.9% average success across four LIBERO suites and 91.10% / 91.24% on RoboTwin Clean / Randomized are reported alongside real-world evaluation protocols.The evaluation targets spatial understanding, long-horizon decision-making, and applicability across manipulation tasks.Source: §1 · p. 2
  • The stated contributions combine geometry-supervised spatial representations, geometry-aware event memory, and evaluation across LIBERO, RoboTwin, and real-world manipulation tasks.The evaluation is framed around geometric precision and the use of historical context in long-horizon manipulation.Source: §1 · p. 3

2 RELATED WORK

Related work spans VLA architectures, geometric supervision, and temporal or spatial memory. The section identifies how to retain and access scene and motion information across decisions as an open question.

Vision-Language-Action Models

  • VLA advances combine pretrained vision-language models with robot-data scaling and varied action decoders, while leaving open how task-relevant 3D structure should be encoded and retained across decisions.Examples include discrete action prediction, continuous action experts, frequency-domain action compression, and parallel decoding with action chunking.Source: §2 · p. 3

3D Representations and Geometric Supervision

  • Geometry-aware VLA work uses depth, feature-level supervision, or predictive representations, but geometric supervision alone does not specify how information is maintained and accessed throughout an episode.The cited approaches include feature alignment, spatial reasoning distillation, future robot-keypoint trajectories, and current and future Gaussian scene prediction.Source: §2 · p. 3

Temporal and Spatial Memory

  • History-aware policies use recurrent states, visual prompts, and retrievable memory, yet how scene and motion targets shape latent history for later decisions remains open.Existing designs trade retained detail against temporal coverage and cost.Source: §2 · p. 3

3 METHOD

GeoMem-VLA predicts action chunks from language-conditioned observations while learning compact geometric tokens from scene and future-motion targets. It retrieves and consolidates geometric and observational history to condition action generation.

3.1 PRELIMINARIES

  • The policy predicts continuous action chunks from visual observations, proprioceptive state, and language using a flow-matching action expert.At inference, Euler integration transforms Gaussian noise into an action chunk.Source: §3.1 · pp. 3–4

3.2 GEOMETRY-SUPERVISED LATENT REPRESENTATIONS

  • Learnable GeoPlan markers produce projected geometric tokens supervised by scene point maps and future Cartesian end-effector tracks.The geometric objective is combined with the action flow-matching objective, with α controlling their relative weight.Source: §3.2 · pp. 4–5

3.3 GEOMETRY EVENT-AWARE MEMORY

  • Causal retrieval uses earlier episode entries, while overflow consolidation averages the most similar temporally adjacent pair to retain feature-distinct states.At deployment, the geometry teacher and decoder are omitted; latent tokens and memory operations remain active.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate GeoMem-VLA on LIBERO and RoboTwin, use component ablations and representation-memory diagnostics, and extend evaluation to five real-world manipulation tasks.

4.1 EXPERIMENTAL SETUP

  • Evaluation covers four 10-task LIBERO suites and 50 RoboTwin bimanual tasks, with 50 trials per LIBERO task and 100 per RoboTwin task and setting.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • 98.9% LIBERO average success (highest displayed, 0.4 points above Spatial Forcing) and 91.10% / 91.24% RoboTwin Clean / Randomized.Source: §4.2 · p. 7

4.3 ABLATION STUDY

  • Removing GeoPlan or temporal memory lowers RoboTwin success in both settings; on LIBERO, the observation-memory variant averages 97.9% versus 98.9% for the full model.The GeoPlan intervention removes its supervision, tokens, geometric bank, and gated fusion; the temporal-memory intervention keeps current geometric tokens but disables historical retrieval.Source: §4.3 · p. 7

4.4 REPRESENTATION AND MEMORY DIAGNOSTICS

  • Across four cases, point-map RMSE is 0.064–0.094 normalized units and EEF RMSE is 3.07–3.74 mm; retained entries span initial layout and later stacking stages.The diagnostic causally processes 16 sampled episode frames and decodes point maps and future EEF tracks at four frames.Source: §4.4 · p. 8, Figure 3 · p. 8

4.5 REAL-WORLD EXPERIMENTS

  • Across five real-world tasks, success is 90% for pick-and-stand sticks, 70% for cup transfer, 80% for covering and finding the apple, and 100% for file-pocket separation.Each Figure 4 bar reports 10 independent closed-loop rollouts.Source: §4.5 · p. 9, Figure 4 · p. 9

5 CONCLUSION

GeoMem-VLA treats geometry-supervised latent memory as a connection between 3D representation learning and action generation over time. The paper evaluates object manipulation in simulation and on physical robots, but not human–robot interaction.

  • The authors present geometry-supervised latent memory as a practical connection between 3D representation learning and action generation over time.Source: §5 · p. 9
  • Evaluation covers object manipulation in simulation and on physical robots, without testing human–robot interaction; reported task success does not establish safety around people or in unconstrained settings.Real-world use requires supervision and appropriate safety controls.Source: ETHICS STATEMENT · p. 10
  • The code is slated for public release upon publication, with additional training and evaluation details in the appendix.Source: REPRODUCIBILITY STATEMENT · p. 10

A IMPLEMENTATION DETAILS AND HYPERPARAMETERS

The implementation uses frozen VGGT supervision with Qwen3-VL-4B, while geometric fusion projects geometry into the visual-token dimension and gates it with visual features. The learned gate initially favors the visual path.

A IMPLEMENTATION DETAILS AND HYPERPARAMETERS

B GEOMETRIC FUSION.

  • Cross-attention projects geometric features to the visual dimension before the action expert combines the resulting representation with retrieved observations and robot state.Source: B GEOMETRIC FUSION · p. 15
  • A learned channel-wise gate blends geometric and visual features, with zero-initialized geometry and an initial preference for visual tokens.Source: B GEOMETRIC FUSION · p. 15

C PER-TASK ROBOTWIN RESULTS

The RoboTwin per-task analysis reports results for three variants across Clean and Randomized settings, using task-adaptive execution horizons. Means are recomputed from 50 per-task scores for each setting.

C PER-TASK ROBOTWIN RESULTS

D CASE-STUDY DETAILS

The case studies report point-map reconstruction errors, describe memory-bank retention, and document the diagnostic and hardware setups. The RoboTwin memory replay tracks overlap between independently consolidated observation and geometric banks.

D CASE-STUDY DETAILS

  • The diagnostics decode point maps and end-effector tracks at four selected frames per episode using 20 Euler steps and a fixed noise seed.Memory visualization resets all banks per episode and retrieves only preceding sampled observations.Source: D CASE-STUDY DETAILS · p. 15
  • Scene error uses 256 point correspondences in centroid-centered, unit-RMS-radius coordinates, while end-effector error uses 100 waypoints across both arms and is converted to millimeters.Source: D CASE-STUDY DETAILS · p. 15
  • 0.032–0.152 normalized units is the per-frame point-map RMSE range across all 16 reconstructed frames.Source: D CASE-STUDY DETAILS · p. 16
  • In the randomized stacking replay, 16 writes produce eight merges per capacity-eight observational bank; four of six shared final observational timestamps also remain in geometric memory.The shared timestamps are 94, 126, 314, and 471.Source: D CASE-STUDY DETAILS · p. 16
  • Figure 5 compares demonstration-derived target and predicted end-effector tracks across 50 waypoints per arm, with Euclidean error shown at each future index.Source: Figure 5 · p. 17

E.1 HARDWARE AND EXPERIMENTAL SETUP

  • The study evaluates GeoMem-VLA on Astribot S1 and AgileX Cobot Magic 2, both dual-arm platforms with three-camera RGB systems.Astribot S1 has two 7-DoF arms; AgileX has two 6-DoF arms.Source: §E.1 · p. 17

E.2 TASK DESCRIPTION

The real-world tasks cover object placement, sequential manipulation, hidden-object reasoning, and repeated actions across two robot platforms. The case-study figures include successful and unsuccessful rollout examples, alongside geometry and memory diagnostics.

E.2 TASK DESCRIPTION

  • Five real-world tasks span precise placement and sequential manipulation with changing task states across different interaction patterns.Source: §E.2 · p. 17
  • Figure 6 visualizes frozen-teacher geometry targets, GeoPlan point maps, predicted future end-effector tracks, memory retrieval attention, and retained-event alignment.Source: Figure 6 · p. 18
  • Figures 7 and 9 show representative successful rollouts, while Figures 8 and 10 show failures on selected Astribot and AgileX tasks.Source: Figure 7 · p. 19, Figure 9 · p. 20, Figure 8 · p. 20
  • Astribot S1 tasks include ordering three colored sticks upright, transferring a cup between grippers across a rack, and selecting and placing a target file pocket.Source: §E.2 · p. 19
  • AgileX tasks require covering and revealing an apple before grasping it, and repeatedly placing a cube exactly the number of times specified.Source: §E.2 · pp. 20–21

E.3 EVALUATION PROTOCOL

Real-world evaluation randomizes initial conditions and task-specific parameters, and counts success only when the full task objective is completed.

E.3 EVALUATION PROTOCOL

  • Object positions and task-specific parameters are randomized before each real-world rollout.Ten independent rollouts are conducted per task.Source: §E.3 · p. 21
  • A rollout succeeds only if the robot completes the entire task and meets the predefined objective; partial completion counts as failure.Source: §E.3 · p. 21
  • Each task’s reported success rate is computed over 10 evaluation rollouts.Source: §E.3 · p. 21

E.4 BASELINE IMPLEMENTATION

The real-world baseline protocol matches task data and rollout conditions across models, while supplemental figures visualize geometry and memory on representative tasks.

E.4 BASELINE IMPLEMENTATION

  • Baselines use the same task-specific demonstrations and are evaluated with identical initial configurations, episodes, and success criteria.LingBot-VLA 2.0, π0.5, and OpenVLA-OFT follow their official configurations and recommended training pipelines.Source: §E.4 · p. 21

E.5 ADDITIONAL QUALITATIVE RESULTS

  • Figures 11 and 12 visualize teacher geometry, GeoPlan point maps, future end-effector waypoints, memory retrieval similarity, and retained-event alignment.The examples cover three Astribot tasks and two AgileX tasks.Source: §E.5 · p. 21, Figure 11 · p. 22, Figure 12 · p. 23
Loading 2610.29866v1…