PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometric Action Model for Robot Policy Learning

Jisang Han, Seonghu Jeon, Jaewoo Jung, René Zurbrügg, Honggyu An, Tifanny Portela, Marco Hutter, Marc Pollefeys, Seungryong Kim, Sunghwan Hong

arXiv:2606.17046v2cs.ROcs.CVcs.LG

TL;DR

Generalist robot policies must integrate language, visual information, scene geometry, robot state, and physical dynamics, but existing foundation-model approaches leave crucial 3D cues implicit. GAM repurposes a geometric foundation model for shared perception, future-state prediction, and action decoding, matching or exceeding foundation-model-scale baselines while running 55× faster and improving camera-perturbation performance by 9.7 percentage points.

  • Problem

    Existing vision-language-action and video world-action models operate fundamentally in 2D, leaving depth, scale, and occlusion implicit for contact-rich manipulation.

  • Method

    GAM inserts a causal future predictor within a pretrained geometric foundation model to jointly decode language-conditioned future geometry and robot actions.

  • Results

    Across simulation and real-world benchmarks, GAM matches or exceeds foundation-model-scale baselines, runs 55× faster, and improves camera-perturbation performance by 9.7%p.

  • Takeaways & Limitations

    GAM demonstrates that a shared geometric substrate can support temporal world modeling and action prediction with strong accuracy, efficiency, and robustness across manipulation benchmarks.

  • Takeaways & Limitations

    The method assumes the split layer is deep enough for rich visual features yet shallower than the earliest DPT-head layer for future-geometry decoding.

Abstract

from arXiv · show

Generalist robot policies must follow user instructions while reasoning about how objects, cameras, and robot actions interact in the 3D physical world. Recent vision-language-action models (VLAs) and video world-action models (WAMs) inherit strong semantic or temporal priors from large-scale foundation models, but they still operate primarily on 2D image frames or 2D-derived latent spaces, leaving implicit the 3D geometry required for contact-rich manipulation. We propose the Geometric Action Model (GAM), a language-conditioned manipulation policy that directly repurposes a pretrained geometric foundation model (GFM) as a shared substrate for perception, temporal prediction, and action decoding. GAM splits the GFM at an intermediate layer: the shallow layers serve as an observation encoder, and a causal future predictor inserted at the split layer forecasts future latent tokens conditioned on language, proprioception, and action history. The predicted future tokens are then routed through the remaining GFM blocks for feature propagation and decoding, allowing a single backbone to produce both future geometry and actions. This design equips the GFM with language-conditioned temporal world modeling through minimal architectural modification while preserving its rich geometric priors. Across a broad suite of simulation and real-robot manipulation benchmarks, GAM is more accurate, more robust, faster, and lighter than current foundation-model-scale baselines.

1 Introduction

GAM is a language-conditioned manipulation policy that repurposes a pretrained geometric foundation model as a shared substrate for perception, future-state prediction, and action decoding. Across simulation and real-world benchmarks, it matches or exceeds foundation-model-scale baselines while being faster, lighter, and more robust to camera perturbations.

  • Generalist manipulation policies must follow natural-language instructions while reasoning about language, visual appearance, scene geometry, robot state, and physical dynamics.
  • VLAs map visual and linguistic tokens to actions, whereas WAMs use pretrained video-generation priors to jointly model future frames and actions.
  • GAM directly repurposes a geometric foundation model as a shared medium for perception, future-state prediction, and action decoding.The model jointly predicts future action and geometry so geometric world dynamics are incorporated into the policy.
  • GAM splits the pretrained geometric foundation model at an intermediate layer, using shallow layers for observation encoding and remaining layers for decoding around a causal future predictor.The predictor uses current visual observations and is conditioned on task language, proprioception, and action history.
  • 55× faster inference and ↑9.7%p camera-perturbation performance highlight GAM’s efficiency and geometric generalization across simulation and real-world benchmarks.GAM matches or exceeds current foundation-model-scale baselines while using substantially fewer trainable parameters and showing improved generalization to unseen scenarios.

2 Related Work

Related work spans vision-language-action policies, world action models that predict future states, and geometric foundation models for robot manipulation. GAM differs by jointly modeling actions and future scenes in one autoregressive sequence and using deep GFM blocks to decode predicted future tokens.

  • Vision-Language-Action Models: VLAs adapt pretrained vision-language models into robot policies that decode actions through autoregressive, parallel, flow-matching, diffusion-based, frequency-space, or compact-VLM approaches.These methods extend large generalist foundation models for robot control.
  • World Action Models for Robot Manipulation: World action models learn policies by predicting future states, using video-generation backbones or separate temporal predictors.One branch jointly predicts future frames and actions, while another keeps the visual backbone frozen and trains a separate temporal predictor.
  • Geometric Foundation Models for Manipulation: Geometric foundation models infer dense 3D structures from multi-view images and have been used as perceptual substrates through feature extraction, encoder replacement, or point-cloud fusion.Recent work also applies GFMs to predictive control, moving beyond static feature extraction.
  • 3D latents: GAM jointly models actions and future-scene predictions in a single autoregressive token sequence and repurposes deep GFM blocks to decode predicted future tokens.This contrasts with separated heads or diffusion processes and with using deep blocks only on observed tokens.

3 Preliminaries: Geometric Foundation Models

Geometric foundation models are feed-forward transformers that infer dense, shared-frame 3D geometry and camera parameters from one or more RGB views. They process patch-based view tokens with frame-wise or cross-view global attention, then decode geometry from selected intermediate layers.

  • GFM inputs and outputs: GFMs map one or more RGB images to dense 3D geometry in a shared world frame, with auxiliary heads predicting camera intrinsics and extrinsics.They can produce per-pixel depth or 3D point maps alongside per-view camera parameters.
  • Tokenization: Each view is partitioned into P non-overlapping p × p patches and projected into a token sequence that includes a per-view camera token.The full input sequence concatenates tokens across views.
  • Attention modes: The transformer stack uses frame-wise attention within each view or global attention across all V(1 + P) tokens to fuse information across viewpoints.Frame-wise attention operates independently over a single image’s tokens, whereas global attention jointly processes all view tokens.
  • Geometry decoding: Dense geometry is decoded from selected intermediate hidden states S = {m1, m2, m3, m4} using a DPT head to estimate per-pixel geometry.The selected multi-layer hidden states are extracted after the transformer forward pass.

4 GAM: Geometric Action Model

GAM transforms a pretrained geometric foundation model into a language-conditioned world-action model by inserting a causal future predictor between shallow encoding and deep decoding. It jointly forecasts future geometric latents and actions in the GFM’s geometric representation space, enabling temporal world modeling with minimal architectural modification.

  • Overview: GAM splits the pretrained GFM at an intermediate layer and inserts a causal temporal predictor, keeping prediction inside the model’s geometric latent space.The shallow layers encode observations, while the remaining blocks propagate and decode predicted future representations.
  • Observation Encoder: The observation encoder reuses shallow GFM layers to convert multi-view RGB observations into per-timestep latent geometric states.The split layer must extract rich visual features while remaining before the earliest DPT-head layer so future geometry remains decodable.
  • Causal Future Predictor: The causal predictor combines language, proprioception, action history, and GFM latents to forecast future geometric tokens without future leakage.Block-causal self-attention processes the context window, conditioning predictions on past and present information only.
  • Causal Future Predictor: GAM jointly predicts future geometry and the next action token at the split layer, tightly coupling action prediction with spatial representations.Only the inserted predictor learns to fuse language, proprioception, action history, and GFM features; its outputs are passed to the remaining GFM blocks.
  • Feature Propagation and Decoding: The remaining GFM blocks propagate per-view action tokens and decode both executable action chunks and action-aligned future depth maps.An action head aggregates action tokens, while the original GFM depth head decodes predicted geometry into future depth.
  • Training and Inference: Training combines action execution, future-feature alignment, and future-depth supervision, while inference uses key-value caching to process each new observation in one feed-forward pass.The objective weights action, feature, and depth losses; depth supervision uses scale-invariant and gradient-matching penalties.

5 Experiments

GAM is evaluated across simulation and real-robot manipulation benchmarks, where it improves robustness to camera perturbations while maintaining competitive standard performance. Ablations and efficiency analyses further examine its components, split-layer choice, latency, and model size.

  • Experimental Setup: GAM is pretrained on 784K single-arm trajectories, then post-trained per benchmark and evaluated against VLAs, WAMs, and geometry-aware VLAs under matched protocols.The real-robot comparisons specifically include π0.5 and Spatial Forcing.
  • Simulation Results: On LIBERO-Plus, GAM consistently outperforms competing baselines, including a 9.7 percentage-point gain in the camera-perturbation setting.Performance on standard LIBERO is highly saturated and remains highly competitive.
  • Real-world Results: In real-robot tasks, GAM substantially outperforms all baselines and remains robust under out-of-domain camera-position perturbations.The evaluation uses four tasks and 20 trials per task split equally between nominal and perturbed environments.
  • Post-training Component Analysis: Pretraining is crucial for LIBERO-Plus robustness, while removing future-prediction losses has minimal impact with a pretrained backbone but still provides geometric supervision without pretraining.The component analysis covers Ldepth and Lfeat on the Object suite.
  • Split Layer Ls Selection: The default split layer Ls = 12 achieves peak performance, validating it as the seam between frame-wise and cross-view attention.Layer 19 remains competitive, whereas inserting the predictor too early degrades performance.
  • Inference Speed and Model Size: GAM requires 6.9 ms per feed-forward pass, runs up to 55× faster than Cosmos Policy, and uses only 1.4B parameters while matching prior accuracy and robustness.Single-pass prediction avoids the multi-step denoising used by diffusion policies.
  • Robustness to Viewpoint and Scene Variation: GAM achieves higher success rates than all baselines at every camera-perturbation difficulty level, including the strongest perturbations.The breakdown is reported for LIBERO-Plus.

6 Conclusion and Limitation … A.1 Pre-training Details

GAM unifies geometric prediction, action prediction, and temporal world modeling within a shared geometric foundation model, achieving strong benchmark performance. The appendix details its training data, evaluation settings, analyses, and pre-training procedures.

  • 6 Conclusion and Limitation: GAM unifies geometry and action prediction with temporal world modeling inside a single shared GFM.A causal transformer between shallow and deep GFM layers autoregressively decodes actions and future geometries.
  • 6 Conclusion and Limitation: Across simulation and real-world benchmarks, GAM achieves superior accuracy, faster inference, and strong out-of-distribution robustness.
  • Appendix: The appendix covers training data, implementation details, simulation and real-world evaluation settings, baseline settings, and inference benchmarking.
  • Appendix: Additional appendix results cover LIBERO, LIBERO-Plus, and RoboCasa simulation benchmarks.
  • Appendix: Additional analyses examine backbone variants, pretraining ablations, split-layer choices, action-token attention, and robustness trends.
  • A Experimental Settings and Reproducibility Details: GAM trains in two stages: end-to-end joint training followed by benchmark-specific fine-tuning on official training sets.The first stage jointly trains the predictor, action head, and GFM backbone on mixed single-arm robot data.
  • A.1 Pre-training Details: Pre-training mixes OXE, MimicGen, and RoboCasa365 at sampling ratios of 72%, 18%, and 10%, respectively.OXE supplies broad real-robot coverage, while MimicGen and RoboCasa365 supply simulation demonstrations with geometric supervision.
  • A.1 Pre-training Details: Datasets are standardized to common observation and action formats, using 224×224 images and two RGB views when available.Training applies random cropping, rotation, and color jitter; evaluation disables these augmentations.

A.2 Simulation Experiments Details

The simulation experiments follow the LIBERO evaluation protocol across four task suites, using filtered demonstrations and separate policies for each suite. Success rate is the primary metric, with standardized randomized-trial evaluation for the original benchmark.

  • Evaluation Protocol: GAM is evaluated on LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long, each containing 10 tasks.The study follows the evaluation protocol established by OpenVLA and OpenVLA-OFT.
  • Training Setup: Training uses filtered LIBERO demonstrations that remove unsuccessful episodes and near-zero-action idle/no-op frames.A separate policy is fine-tuned for each LIBERO suite.
  • Evaluation Protocol: Task execution success rate (SR, %) is the primary evaluation metric.For the original LIBERO benchmark, each task uses 50 randomized trials, totaling 500 rollouts per suite.
  • Baseline Comparison: The study re-evaluates π0.5, Cosmos-Policy, π0.5 + Spatial Forcing, and π0.5 + ROCKET under its evaluation setting using available or reproduced checkpoints.Results for other baselines are taken from Fei et al. and Zheng et al.

A.3 Real-World Experiments Details … B Additional Benchmark Results

The paper details real-world evaluation tasks, matched baseline protocols, inference-speed comparisons, and GAM’s compact parameterization. It also identifies the benchmark reporting format for additional RoboCasa-Kitchen results.

  • A.3 Real-World Experiments Details: Real-world evaluations use a wrist-mounted ZED camera and an external RealSense camera for third-person observation.The setup is illustrated in Figure 7.
  • A.3 Real-World Experiments Details: Four tasks—Pick and place, Stack milk and cube, Place pot and pan on cooktop, and Insert cube into covered pot—were jointly trained from 284, 202, 184, and 169 demonstrations.Each task used teleoperated demonstrations within a unified dataset.
  • A.3 Real-World Experiments Details: Each real-world task was evaluated over 20 trials, split between 10 normal trials and 10 camera-perturbed OOD trials using 85 cm translation and 45° rotation.The perturbation was applied to the external camera.
  • A.4 Real-World Experiments Baseline Training Details: π0.5 and Spatial Forcing use the same two 224 × 224 RGB inputs as GAM while retaining their original inference protocols.π0.5 uses flow-matching action decoding with 10 integration steps, whereas Spatial Forcing uses its feature alignment loss recipe.
  • A.5 Inference Latency Comparison Details: 17.5 ms is GAM’s matched single-feed-forward action-prediction latency, compared with 29.2 ms for π0.5 and 70.1 ms for OpenVLA-OFT.All policies use Torch Compile without CUDA Graphs on the matched setting.
  • A.6 Model Size Breakdown: GAM uses a 1.4B-parameter model, smaller than VLM- and video-diffusion-based baselines including π0.5, OpenVLA-OFT, and Cosmos-Policy.The model repurposes a pretrained geometric backbone for perception, future-geometry prediction, and action decoding.
  • A.6 Model Size Breakdown: Approximately 983.2M parameters are trainable, while initial geometric layers and the DPT depth head remain frozen.The later ViT-Giant blocks provide most trainable parameters, and the Causal Future Predictor and lightweight action head are fully trainable.
  • B Additional Benchmark Results: Additional benchmark results report per-task success rates and the overall average on RoboCasa-Kitchen.These results are presented in Table 11.

B.1 Additional Results on RoboCasa-kitchen

RoboCasa-Kitchen evaluates GAM on 24 diverse kitchen manipulation tasks in realistic simulation environments. GAM is adapted from its base checkpoint to the benchmark’s three-view, longer-horizon setting using successful demonstration trajectories.

  • Benchmark: RoboCasa-Kitchen covers 24 tasks spanning pick-and-place, articulated-object interaction, appliance control, and coffee-related manipulation.The benchmark is derived from RoboCasa and targets everyday manipulation in realistic, diverse kitchen environments.
  • Adaptation: GAM is further trained from the base pre-training checkpoint to accommodate RoboCasa-Kitchen’s 3-view observations instead of its original 2-view setup.This adapts the policy to the benchmark’s observation format.
  • Adaptation: The action chunk size increases from 8 to 16 steps to better support RoboCasa-Kitchen’s longer-horizon tasks.The benchmark training configuration also re-extracts depth for 300 demonstrations per task and uses only successful trajectories.

B.2 Additional LIBERO and LIBERO-Plus Results … C.1 When to Predict Actions?

GAM maintains strong performance on original LIBERO while improving LIBERO-Plus robustness, particularly for camera-viewpoint perturbations. It also predicts future depth aligned with actions, and passing action tokens through the deep geometric decoder further improves performance under camera perturbations.

  • B.2 Additional LIBERO and LIBERO-Plus Results: LIBERO-Plus breakdowns evaluate GAM across four task suites and perturbation difficulty levels L1–L5, while Table 13 reports task-wise success rates on original LIBERO.These analyses expand the aggregate LIBERO and LIBERO-Plus results with suite-, perturbation-, difficulty-, and task-level evaluations.
  • B.2 Additional LIBERO and LIBERO-Plus Results: GAM preserves strong original-LIBERO performance while improving LIBERO-Plus robustness, especially for perturbations requiring stable geometric understanding such as camera-viewpoint changes.The robustness analysis emphasizes behavior as perturbation severity increases.
  • B.3 Generated Future Depth Maps: Given a current RGB observation, GAM predicts future depth maps while simultaneously generating actions spatially aligned with anticipated future geometry.The model’s visualizations show corresponding depth forecasts and action sequences across LIBERO task suites.
  • B.3 Generated Future Depth Maps: The visualizations demonstrate that GAM accurately forecasts future depth alongside its corresponding action sequence.This diagnostic links predicted geometry to the actions produced for representative LIBERO tasks.
  • C Ablation and Diagnostic Analyses: The ablation directly supervises the causal future predictor’s output action token without passing it through the remaining DA3 blocks, using the main LIBERO-Object component-ablation setting.This isolates whether the deep geometric decoder contributes to action prediction.
  • C.1 When to Predict Actions?: The direct-action comparison suggests that decoding actions through the remaining geometric layers is preferable to applying action loss directly at the causal predictor output.The decoder’s contribution is most evident for LIBERO-Plus Object and camera-perturbed conditions.
  • C.1 When to Predict Actions?: Passing the action token through the deep geometric decoder provides an additional improvement, particularly on LIBERO-Plus Object.The result indicates that the remaining GFM layers refine the action representation, especially under camera perturbations.

C.2 Attention Analysis

Attention analysis shows that intermediate GFM layers focus on task-relevant visual regions, especially manipulated objects and nearby contact areas. This pattern supports using mid-level representations for action decoding.

  • C.2 Attention Analysis: Intermediate GFM layers attend to task-relevant regions, with saliency around manipulated objects and nearby contact areas.This qualitative attention pattern is consistent with layer ablation: mid-level representations preserve object-level structure while retaining decoder depth for action-token decoding.
Loading 2606.17046v2…