PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounded Action Model: 3D Grounding as a Foundation for Robotics

Gehao Zhang, Weikai Huang, Shailesh Shailesh, Yiyan Peng, Jiafei Duan, Ranjay Krishna

arXiv:2609.23863v1cs.RO

TL;DR

Pretrained language and video backbones do not directly require metric grounding of task-relevant objects for manipulation. GAM uses promptable 3D grounding to condition action prediction on object-centric visual and geometric representations, and reports results across simulation and real robots, including robustness to visual shifts.

  • Problem

    Language- and video-generation pretraining do not directly require the explicit metric grounding needed to identify task-relevant objects and expose their geometry for manipulation.

    Source: §I · p. 1

  • Method

    GAM maps language, point, or box prompts through a pretrained 3D grounding backbone into object-centric visual and metric representations used with robot state history for action prediction.

    Source: §I · p. 1, B. From detections to observation · p. 4

  • Results

    Across simulation and real-robot evaluations, GAM reports stronger performance under scene variation and target changes, including 17/20 successes under visual shift on a bimanual YAM versus 4/20 for π0.5.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results support 3D grounding as a promising foundation for manipulation policies that must respond to task-relevant spatial changes while tolerating visual variation.

    Source: §V · p. 7

    Grounding errors propagate to actions without recovery, and object-centric filtering can omit relevant context such as unselected obstacles.

    Source: §V · p. 7

Abstract

from arXiv · show

Manipulation policies must know which objects matter and where they are, yet the pretrained backbones that current robot foundation models build on, from language in vision-language-action models (VLAs) to video generation in world-action models (WAMs), do not directly require this metric grounding, leaving it to be learned implicitly from robot demonstrations. We propose Grounded Action Models (GAMs), a new paradigm of robot foundation models built with 3D grounding. GAM can be conditioned using language, points, or box prompts, which are first transformed into a shared object-centric representation of the selected objects. This representation captures target-focused visual features and metric object geometry, which is mixed with robot state history through a multi-stream transformer to predict action chunks. Although GAMs can be run autonomously, they can also serve as a low-level controller that a high-level planner controls using its various input modalities, allowing for long-horizon and memory-dependent manipulation. On RoboTwin 2.0, GAM achieves an average success rate of 55.3% across 50 tasks (vs. 52.0% for Spatial Forcing), including 47.6% under scene randomization (vs. 30.4% for Abot-M0), with its action policy trained only on clean-scene demonstrations. On LIBERO-PRO, it achieves a state-of-the-art average success rate of 61% (vs. 53% for $π_{0.5}$) across 16 perturbation settings, with the largest gains when targets are relocated or newly designated. On two real robots, GAM retains 17/20 successes under visual shift on a bimanual YAM versus 4/20 for $π_{0.5}$, while its composition with a Molmo2 planner on a Franka achieves 64.7% ID and 49.8% OOD step completion on long-horizon and memory-dependent tasks.

I. INTRODUCTION

GAM builds robot action learning on promptable 3D grounding, explicitly representing task-relevant objects and their metric geometry. Its evaluations span simulation and real robots, including visual variation and target changes.

I. INTRODUCTION

  • Language- and video-generation objectives do not directly require explicit metric grounding, so demonstrations must connect pretrained representations to spatially precise motor behavior.With limited demonstration coverage, the learned connection can depend on visual correlations that fail when objects move, backgrounds change, or targets change.Source: §I · p. 1
  • GAM uses a pretrained promptable 3D grounding model to expose which objects matter, their locations, and their geometry to the action learner.The grounding backbone supplies localization and spatial understanding while robot demonstrations teach the policy how to act on grounded objects.Source: §I · p. 1
  • GAM retains visual features of selected objects and the robot arm alongside metric 3D positions and extents, then combines these streams with robot state history to predict joint-position chunks.The grounding backbone remains frozen during policy training.Source: §I · p. 1
  • Language, 2D points, and 2D boxes form a unified task interface, allowing visual prompts to directly specify the object for manipulation.A planner can update object selections during multi-step tasks and use observation history when identities or desired locations are not recoverable from the current image.Source: §I · pp. 1–2
  • 55.3% average success across 50 RoboTwin 2.0 tasks (vs. 52.0% for Spatial Forcing), with 47.6% under scene randomization (vs. 30.4% for Abot-M0), using clean-scene demonstrations.On LIBERO-PRO, GAM averages 0.61 versus 0.53 for π0.5 across 16 perturbation settings, with largest gains for relocated or newly designated targets; on a YAM under visual shift it succeeds in 17/20 trials versus 4/20 for π0.5.Source: §I · p. 2
  • The authors present 3D grounding as a promising foundation for policies that respond to task-relevant spatial changes while tolerating visual variation.GAM is instantiated with WildDet3D, originally trained for 3D box detection, while alternative grounding models are anticipated.Source: §I · p. 2

II. RELATED WORK

Prior robot foundation models transfer language or video pretraining to action learning, while other work incorporates explicit geometry and selective perception. GAM investigates promptable 3D grounding as a unified foundation for these spatially grounded observations.

II. RELATED WORK

  • VLAs transfer pretrained vision-language representations to action prediction, while video-action approaches use future visual representations and spatiotemporal priors for control.GAM instead investigates promptable 3D grounding, whose frozen backbone supplies object localization, visual features, and metric geometry for action learning.Source: §II · p. 2
  • Prior manipulation work uses voxel or point-cloud representations, geometry-aware VLAs, selective visual representations, spatial value maps, affordances, and visual guidance.GAM unifies language, point, and box prompts through one promptable 3D grounding backbone and grounded observation.Source: §II · p. 2
  • GAM’s architecture resolves language, point, or box prompts into per-object detections and depth, then combines object-centric image and detection tokens with state history for action prediction.A multi-stream transformer predicts joint-position chunks from these tokens and a language embedding.Source: Figure 2 · p. 3

III. GAM: GROUNDED ACTION MODEL

GAM pairs a frozen promptable 3D grounding backbone with an action head that uses object visual and geometric representations and robot state history. Language, points, and boxes identify task objects through a common interface.

III. GAM: GROUNDED ACTION MODEL

  • The frozen grounding backbone localizes task objects, and the action head combines their visual and geometric representations with robot state history and language to predict joint-position chunks.The policy is built on WildDet3D, which produces 2D and metric 3D boxes, a metric depth map, and dense visual features from image and task prompt.Source: §III · p. 3, A. From prompt to detections · p. 3

A. From prompt to detections

  • Language, 2D points, and 2D boxes identify task objects, which are resolved into per-object 2D detections, metric 3D boxes, and a dense metric depth map.For language input, a span-tagging head extracts task-relevant object phrases as separate grounding queries; point and box prompts specify objects directly.Source: A. From prompt to detections · p. 3
  • The three prompt modalities map to a shared object-centric representation, allowing a VLM planner to specify target objects through points while language separately specifies the operation.Source: A. From prompt to detections · p. 3

B. From detections to observation

GAM forms an observation from task-focused image features, metric object geometry, and robot state history. These representations are fused while retaining modality-specific token streams for action prediction.

B. From detections to observation

  • The observation combines image and detection tokens with a state-history token representing current and recent joint positions.Learned modality embeddings and a shared self-attention layer fuse the tokens, while preserving their modality-specific positions for the action transformer.Source: B. From detections to observation · pp. 3–4
  • Image tokens retain visual features in grid cells overlapping task-object boxes or the robot arm, while zeroing features outside those regions.This preserves selected-object appearance and its spatial relationship to the gripper while suppressing other scene features.Source: B. From detections to observation · p. 3
  • Metric object geometry is constructed by back-projecting depth into a scene point cloud, cropping points inside each predicted 3D box, and sampling 512 points per object.The resulting point cloud is encoded as a shape feature, while the box supplies the object’s center, extents, and rotation in the robot base frame.Source: B. From detections to observation · pp. 3–4
  • Detection tokens expose each selected object’s estimated shape, location, extent, and orientation through eight tokens per object.The descriptor includes normalized 2D box, encoded center, shape feature, extents, and rotation.Source: B. From detections to observation · p. 4

C. From observation to actions

GAM’s multi-stream action head combines grounded visual and geometric observations with robot history and language conditioning to predict action chunks. It trains the head with flow matching and generates actions by integrating the predicted velocity.

C. From observation to actions

  • The 12-block multi-stream transformer predicts chunks of absolute joint-position targets and gripper commands from image, detection, state-history, and noisy-action streams.Joint attention lets information pass among all four streams.Source: C. From observation to actions · p. 4
  • A learned projection of the instruction embedding conditions each transformer block through adaptive layer normalization, adding operation semantics without extra tokens.The projection is added to the flow-timestep embedding; its weights and bias start at zero.Source: C. From observation to actions · p. 4
  • Flow matching trains the action head to predict the velocity from a noisy demonstration chunk, and inference integrates that velocity from Gaussian noise using K Euler steps.The predicted chunk contains absolute joint-position targets and gripper commands.Source: C. From observation to actions · p. 4

D. Deployment

At episode start, GAM grounds objects from language or accepts point or box prompts, then tracks them across frames. For long-horizon tasks, a planner can use observations and episode memory to select new targets while the action policy stays unchanged.

D. Deployment

  • GAM grounds language-parsed objects or accepts point and box prompts, then tracks detections across frames to preserve object associations.CoTracker3 follows points from the initial detection boxes, and the grounded observation is refreshed at each policy query.Source: D. Deployment · p. 4
  • At each query, the action head predicts a chunk from the current grounded observation and instruction embedding, and executes a prescribed number of actions before querying again.Source: D. Deployment · p. 4
  • At subtask transitions, a planner can use current images and episode memory to select new targets and reinitialize their trackers, while the action policy remains unchanged.This supports target selection informed by past observations on long-horizon and memory-dependent tasks.Source: D. Deployment · p. 4

IV. EXPERIMENTS

The experiments test whether promptable 3D grounding improves generalization to randomized scenes. On RoboTwin 2.0, GAM has the highest average success among listed methods and the strongest randomized-scene result.

A. (Q1) Manipulation performance on RoboTwin 2.0

  • Each task uses a separate policy trained on 50 clean-scene demonstrations, with 100 rollouts per task in both clean and randomized settings.The grounding backbone is adapted to the rendered domain and frozen; no randomized scenes are seen during action-policy training.Source: A. (Q1) Manipulation performance on… · pp. 4–5
  • 55.3% average success across RoboTwin 2.0’s 50 tasks exceeds the strongest same-protocol baseline, π0.5 at 45.0%; on randomized scenes, GAM reaches 47.6% versus 30.4% for the next-best entry.The reported Table I results cover clean and randomized settings.Source: Table I · p. 5, A. (Q1) Manipulation performance on… · p. 5
  • Unlike a policy using the full scene or raw point clouds, GAM conditions on grounded task objects, so distractors, background, and appearance changes largely do not enter its observation.The passage notes that DP3, despite using 3D point clouds, drops from 55.2% in clean scenes to 5.0% under randomization.Source: A. (Q1) Manipulation performance on… · p. 5

B. (Q2) Generalization to test-time perturbations on LIBERO-PRO

LIBERO-PRO tests policies trained on fixed layouts under changes to objects, placement, instructions, and target or goal. GAM leads on average across the 16 settings, with particularly strong results when targets move or are newly designated.

B. (Q2) Generalization to test-time perturbations on LIBERO-PRO

  • The protocol trains one GAM policy on LIBERO demonstrations from all four suites, then tests four perturbations: object appearance and size, placement, instruction phrasing, and target or goal.The training data has a fixed object layout for every task.Source: B. (Q2) Generalization to test-time… · p. 5
  • Under Obj, GAM trails on all four suites, especially LIBERO-Goal (0.69) and LIBERO-10 (0.50), where object-size changes may require unseen grasp adjustments.Source: B. (Q2) Generalization to test-time… · p. 5
  • Under Pos, GAM reaches 0.60 on Spatial and 0.47 on Object—two to three times the best baseline—except Goal-Pos, where π0.5 leads, 0.38 to 0.19.Source: B. (Q2) Generalization to test-time… · p. 5
  • 0.61 average across 16 LIBERO-PRO settings exceeds π0.5’s 0.53; GAM is particularly strong on placement and new-target perturbations but trails under object appearance and size changes.Baselines remain strong under Obj and Sem but collapse under Pos and Task, which alter layout or goal.Source: B. (Q2) Generalization to test-time… · p. 5, Table II · p. 6
  • Under Task, every baseline stays below 0.11 on all four suites, while GAM reaches 0.88 on Spatial, 0.50 on Object, and 0.16 on Goal.Task designates an object that appeared only as background in the original task; grounding changes the detection tokens to reflect the new target.Source: B. (Q2) Generalization to test-time… · p. 6

C. (Q3) Real-robot manipulation

Real-robot evaluations test direct target specification on a bimanual YAM and planner-driven long-horizon and memory-dependent tasks on a Franka. GAM performs better under visual or distribution shift in both settings.

YAM

  • The YAM evaluation uses four target combinations, 25 demonstrations per combination, and 20 trials in each condition, without a planner or episode memory.GAM receives direct click or 2D-box targets; π0.5 uses its language interface.Source: C. (Q3) Real-robot manipulation · p. 6

Franka

  • The Franka evaluation covers two long-horizon and two memory-dependent tasks, with a Molmo2 planner selecting targets from current observations and episode history.Methods are fine-tuned on 50 demonstrations per task; OOD trials vary backgrounds, lighting, object instances and colors, and add distractors.Source: C. (Q3) Real-robot manipulation · p. 6

YAM

  • 17/20 successes under visual shift for GAM versus 4/20 for π0.5 on the YAM, after both achieve 19/20 in-distribution.GAM and π0.5 retain 89% and 21% of their respective in-distribution success rates.Source: C. (Q3) Real-robot manipulation · p. 6

Franka

  • 64.7% ID and 49.8% OOD step completion rates for GAM–Molmo2, versus 42.4%/24.0% for π0.5 and 35.3%/17.1% for MolmoAct2.GAM retains 77% of its ID performance under shift, compared with 57% for π0.5 and 48% for MolmoAct2.Source: C. (Q3) Real-robot manipulation · p. 6

D. (Q4) Ablations

Ablations show that object-centric filtering and the combination of visual appearance with metric geometry matter for success. Attention examples also show GAM remaining focused on selected targets under distribution shift.

Attention under distribution shift

  • In the shown Clear the table examples, GAM’s attention stays concentrated around selected targets under shift, while baseline attention spreads more over perturbed backgrounds and distractors.The maps compare in-distribution examples (top) with out-of-distribution examples (bottom) and are min-max normalized per image.Source: Figure 4 · p. 7

Object-centric filtering

  • 22.3% success without image masking and 12.8% without point cropping, versus 46.8% for the full observation; removing both filters yields 10.3%.The comparison keeps the backbone, action head, training data, and schedule fixed.Source: D. (Q4) Ablations · pp. 6–7, Table IV · p. 7
  • 17.0% Easy and 3.5% Hard success without either filter, versus 51.5% and 42.0%; the larger Hard degradation supports excluding irrelevant scene content under randomization.The evaluation uses 10 RoboTwin 2.0 tasks with 20 rollouts per task in Easy and Hard.Source: D. (Q4) Ablations · pp. 6–7

Appearance and metric geometry

  • 16.0% success with detection tokens only and 20.3% with image tokens only, versus 46.8% for the full model, which combines metric geometry and appearance.Both single-stream variants retain the state-history token; neither matches the full model on Easy or Hard.Source: D. (Q4) Ablations · p. 7

V. CONCLUSION

GAM connects task specification to action prediction through a pretrained promptable 3D detector, using object-centric visual and metric representations for manipulation. Its grounding and filtering also create limits tied to observation quality, completeness, and geometric representation.

Conclusion

  • GAM connects task specification to action prediction through a pretrained promptable 3D detector, with object-centric visual and metric representations supporting robust manipulation and target generalization.Its unified interface supports direct human target selection and planner-driven long-horizon and memory-dependent manipulation on real robots.Source: §V · p. 7

Limitations and future work

  • GAM depends on grounded-observation quality and completeness: grounding errors propagate to actions without recovery, and filtering can omit unselected obstacles.The paper identifies more diverse 3D grounding data and models that generalize across objects and scenes as areas for improvement.Source: §V · p. 7
  • The 3D bounding boxes used here are one grounding option; dense 3D instance segmentation may provide finer-grained geometric supervision.The authors also propose exploring grounding-aware failure recovery and adaptive context selection for safe execution.Source: §V · p. 8

Related papers

Loading 2609.23863v1…