PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation

Haoquan Fang, Markus Grotz, Wilbert Pumacay, Yi Ru Wang, Dieter Fox, Ranjay Krishna, Jiafei Duan

arXiv:2501.18564v4cs.RO

TL;DR

Robotic manipulation still lacks reliable generalization and spatial memory in diverse environments. This paper introduces SAM2Act, SAM2Act+, and MemoryBench, achieving strong benchmark performance while retaining stated limits in control dexterity, memory-window adaptability, and semantic storage.

  • Problem

    Robotic manipulation methods still have limited evidence for robust multitask performance, generalization to novel environments, and episodic recall.

  • Method

    The paper develops SAM2Act, a multi-view policy combining multi-resolution upsampling with visual embeddings from a large-scale foundation model, then extends it to SAM2Act+ with memory components.

  • Results

    SAM2Act+ achieves 94.3% average success across MemoryBench tasks, while SAM2Act achieves 86.8%±0.5 average success across 18 RLBench tasks and a 4.3% average decrease under Colosseum perturbations.

  • Takeaways & Limitations

    MemoryBench provides a systematic evaluation of spatial memory, and SAM2Act+ demonstrates strong performance across simulated and real-world manipulation settings.

  • Takeaways & Limitations

    SAM2Act+ remains limited in dexterous continuous control, task-varying memory-window length, and retaining semantic information such as color.

Abstract

from arXiv · show

Robotic manipulation systems operating in diverse, dynamic environments must exhibit three critical abilities: multitask interaction, generalization to unseen scenarios, and spatial memory. While significant progress has been made in robotic manipulation, existing approaches often fall short in generalization to complex environmental variations and addressing memory-dependent tasks. To bridge this gap, we introduce SAM2Act, a multi-view robotic transformer-based policy that leverages multi-resolution upsampling with visual representations from large-scale foundation model. SAM2Act achieves a state-of-the-art average success rate of 86.8% across 18 tasks in the RLBench benchmark, and demonstrates robust generalization on The Colosseum benchmark, with only a 4.3% performance gap under diverse environmental perturbations. Building on this foundation, we propose SAM2Act+, a memory-based architecture inspired by SAM2, which incorporates a memory bank, an encoder, and an attention mechanism to enhance spatial memory. To address the need for evaluating memory-dependent tasks, we introduce MemoryBench, a novel benchmark designed to assess spatial memory and action recall in robotic manipulation. SAM2Act+ achieves an average success rate of 94.3% on memory-based tasks in MemoryBench, significantly outperforming existing approaches and pushing the boundaries of memory-based robotic systems. Project page: sam2act.github.io.

1 Introduction

Robotic manipulation requires multitask interaction, generalization to unseen environments, and memory, but prior methods leave challenges in multitask performance, novel configurations, and episodic recall. SAM2Act and SAM2Act+ address these gaps with foundation-model visual features, memory architecture, and MemoryBench evaluation.

  • Diverse robotic environments require multitask interaction, generalization to unseen scenarios, and retention of past experiences.
  • Prior approaches improved 3D manipulation but still face challenges in multitask performance, generalization to novel configurations, and episodic recall.
  • SAM2Act integrates multi-resolution upsampling with visual embeddings from large-scale foundation models in a multi-view robotics transformer policy.
  • SAM2Act+ adds a memory bank, encoder, and attention mechanism to enable episodic recall for spatial memory-dependent manipulation tasks.
  • MemoryBench evaluates spatial memory and the ability to retain and recall past actions in behavior-cloning policies.
  • 94.3% average success rate across MemoryBench tasks was reported for SAM2Act+, while SAM2Act was evaluated for generalization on The Colosseum.

2 Related Work

Earlier manipulation methods progressed from efficient 2D representations to more precise 3D and multi-view approaches, while visual foundation models increasingly supported robust policy learning. Robotic memory research spans maps, neural representations, Gaussian splats, and relational memory, motivating SAM2Act’s visual and memory-based design.

  • 2D manipulation methods offer fast training and low computational cost but struggle with high precision, robust spatial interaction, and environmental or camera variation.
  • 3D methods use point clouds, voxels, or multiscale features, while RVT and RVT-2 use multi-view representations and coarse-to-fine precision improvements.
  • SAM2Act combines RVT-2 spatial reasoning with enhanced virtual images from the SAM2 visual encoder for precision and generalization.
  • Visual representations from pretraining, co-training, and frozen encoders support policy learning, invariance, equivariance, and out-of-distribution generalization.
  • Robotic memory research includes semantic maps, cognitive architectures, VLM- or LVM-derived fields, Gaussian splats, and transformer-based relational memory.

3 MemoryBench: A Memory Benchmark for Robotic Manipulation

MemoryBench is designed around manipulation tasks in which identical current observations can require different actions because the correct response depends on prior action history. Its benchmark includes simulated spatial-memory tasks and evaluations of SAM2Act+ in simulation and real-world settings.

  • Benchmark purpose: MemoryBench systematically evaluates the spatial memory capabilities of robotic manipulation policies.
  • Task Design: Its tasks violate the Markov assumption by making subsequent actions depend on action history rather than only the current observation and action.
  • Evaluation settings: Figure 2 contrasts three simulated benchmark tasks with four Franka Panda real-world tasks, including out-of-distribution perturbation tests.
  • Task Design: Standardized language instructions prevent spatial information from leaking into task descriptions and aiding memory-based decisions.
  • Benchmark tasks: MemoryBench extends RLBench with three scripted spatial-memory tasks: reopen_drawer, put_block_back, and rearrange_block.
  • Benchmark tasks: The tasks contain two to four variations and additional intermediate steps that disrupt the Markov property and require memory.
  • Benchmark tasks: reopen_drawer tests recalling which drawer was initially open after all drawers become visually indistinguishable.
  • Benchmark tasks: rearrange_block tests backward spatial reasoning by requiring the agent to infer which block to move from prior placements and actions.

4 Method

SAM2Act combines multi-view 3D scene reasoning with SAM2 visual embeddings and multi-resolution upsampling for precise action prediction. SAM2Act+ extends this architecture with memory components that condition manipulation on prior actions.

  • SAM2Act Architecture: SAM2Act reconstructs scene point clouds, renders orthogonal virtual views, and uses a coarse-to-fine multi-view transformer to predict action heatmaps.
  • SAM2Act Architecture: SAM2 image embeddings are combined with virtual-image RGB, depth, 3D translation coordinates, and language instructions before multi-view transformer processing.
  • SAM2Act Architecture: The coarse SAM2Act branch predicts translation, while the fine branch generates additional features for actions beyond translation.
  • Multi-Resolution Upsampling: Multi-resolution upsampling integrates SAM2 embeddings through element-wise addition and layer normalization while doubling spatial dimensions at each stage.A cascade of three convex upsamplers progressively refines features for translation heatmap generation.
  • SAM2Act+: SAM2Act+ integrates Memory Attention, Memory Encoder, and Memory Bank into the coarse branch to retain prior actions for spatial memory tasks.The architecture is a task-specific extension of SAM2Act inspired by SAM2.
  • SAM2Act+: SAM2Act+ fine-tunes the coarse branch with memory components and multi-resolution upsampling, using consecutive action keyframes to predict translations conditioned on memory.The SAM2 image encoder, coarse-branch multi-view transformer, and fine branch are frozen during this training setup.

5 Experiments

The experiments evaluate SAM2Act and SAM2Act+ across simulated and real-world manipulation settings, including multitask performance, perturbation robustness, and spatial-memory tasks. SAM2Act performs strongly on RLBench and The Colosseum, while SAM2Act+ improves memory-task performance on MemoryBench and both methods show real-world applicability.

  • 5.2 Performances Across 18 RLBench Tasks: 86.8%±0.5 average success rate on 18 RLBench tasks makes SAM2Act the leading policy, surpassing RVT-2 by 5.4%.SAM2Act ranks first in 9 of 18 tasks and remains within one successful attempt or 4% of the best result in 7 others.
  • 5.2 Performances Across 18 RLBench Tasks: 44% improvement over RVT-2 on Insert Peg and 29% on Sort Shape show SAM2Act’s strongest gains on precise manipulation tasks.The paper attributes these gains to SAM2Act’s multi-resolution upsampling strategy.
  • 5.2 Performances Across 18 RLBench Tasks: 5.8% margin over RVT-2 is reported for SAM2Act in the RLBench multi-task comparison.The table evaluates 18 tasks across 3D keyframe-based behavior-cloning policies using four evaluations for SAM2Act.
  • 5.3 Semantic Generalization across Tasks: 4.3% average performance decrease under 13 perturbation categories is the smallest drop among the evaluated baselines on The Colosseum.SAM2Act is particularly robust to lighting, table appearance, distractors, and camera-pose changes, while remaining competitive under object-level perturbations.
  • 5.4 Performance on MemoryBench: 94.3% average success rate across MemoryBench tasks demonstrates SAM2Act+’s spatial-memory performance, while SAM2Act without memory is 39.3% lower.MemoryBench evaluates three tasks designed to require semantic and spatial memory of past scenes and actions.
  • 5.5 Real-robot Evaluations: 75% real-world task success for the method exceeds RVT-2’s 43%, including 60% versus 0% on high-precision tasks.On the memory-based Push the same button task, SAM2Act achieves 70% success compared with RVT-2’s 40%.

6 Conclusion & Limitation

SAM2Act provides a multi-view, language-conditioned behavior cloning policy for precise 6-DoF manipulation and generalization, while SAM2Act+ adds spatial memory for memory-dependent tasks. The approach remains limited by fixed memory windows, weak semantic retention, and unaddressed dexterous continuous control.

  • SAM2Act is a multi-view, language-conditioned behavior cloning policy for high-precision 6-DoF 3D manipulation that generalizes to unseen perturbations.
  • SAM2Act+ extends SAM2Act with spatial memory to solve spatial memory-based manipulation tasks.
  • SAM2Act and SAM2Act+ achieve state-of-the-art performance across multiple benchmarks.
  • SAM2Act+ remains limited by a task-dependent fixed memory window, restricted semantic retention, and unaddressed dexterous continuous control.

A Model Architecture

The architecture combines multi-view transformer processing with memory attention, a memory encoder, and per-view memory banks. These components condition current observations on past observations and predicted translations while constructing and storing spatial memory features.

  • Multi-View Transformer: The multi-view transformer receives task language, virtual scene-point-cloud images, and SAM2 image embeddings as inputs.
  • Memory Attention: Memory attention conditions current observation features on past observation features and predicted translation actions.
  • Memory Encoder: The memory encoder downsamples translation heatmaps, combines them with unconditioned observation embeddings, and integrates the result through lightweight convolutions.
  • Memory Bank: Each view maintains an independent FIFO memory bank containing up to N recent spatial feature-map memories projected to dimension 64.

B Training Implementation

Training uses RVT2-style augmentation and pipelines for SAM2Act, while SAM2Act+ requires spatio-temporally consistent consecutive observations and a staged pretraining and fine-tuning strategy. Evaluation and training configurations use task-specific or universal hyperparameter procedures across RLBench, The Colosseum, and MemoryBench.

  • Training Resources: Training uses 32 NVIDIA H100/A100 GPUs in the standard configuration while preserving total batch size across alternative GPU counts.
  • Training Pipeline: SAM2Act training follows RVT2 data augmentation and training procedures on RLBench and The Colosseum.
  • SAM2Act+ Training: SAM2Act+ samples n consecutive observations to provide spatio-temporally consistent batches for conditioning actions on current and past observations.
  • SAM2Act+ Training: 20: Pretraining with independent observations followed by fine-tuning on consecutive observations mitigates slow convergence caused by reduced batch diversity.
  • MemoryBench Training: MemoryBench uses universal training hyperparameters, evaluates all epochs, and selects the best-performing checkpoint across tasks.

D Full Results for The Colosseum

The Colosseum results report SAM2Act performance under varied perturbations, while the surrounding evaluation tables document full benchmark comparisons and MemoryBench training configurations. The supplied passages identify the result tables and their evaluation protocols but do not state the Colosseum outcome itself.

  • The Colosseum Results: Table 8 contains the full SAM2Act results for the perturbation settings evaluated on The Colosseum.
  • Benchmark Context: MemoryBench training uses maximum window sizes of 8 for reopen_drawer and 10 for the other two tasks.
  • Benchmark Context: RLBench comparison results cover 18 tasks and show SAM2Act outperforming all baselines by 1.9% over ARP+.
  • The Colosseum Results: The Colosseum evaluation reports mean and standard deviation across three evaluations for varied perturbations.

E.1 RLBench

SAM2Act’s RLBench ablations show that its SAM2 encoder and multi-resolution upsampling components each contribute to performance, with the full model outperforming tested variants.

  • Comparison Setup: The SAM-E action-sequence policy was incompatible with the coarse-to-fine pipeline and converged very slowly under its original training setup.The authors extended SAM-E toward the RVT2 backbone for comparison and retained its original hyperparameters for fairness.
  • RLBench Ablations: All SAM2Act variants underperform the original model on RLBench, whose reported average success rate and ranking are summarized in Table 9.The table reports success rates for 18 RLBench tasks and compares the average success rate and ranking across variants.
  • RLBench Ablations: 1.1% lower average success follows removal of SAM2’s multi-resolution image-embedding inputs.
  • RLBench Ablations: 2.6% lower average success follows replacing multi-resolution upsampling with RVT-2’s original convex upsampling.
  • RLBench Ablations: 6.0% lower performance follows replacing the SAM2 image encoder with the SAM image encoder, while DINOv2 and Depth Anything V2 also show large drops.The SAM comparison is against SAM2Act; the passage also reports a 3.4% drop relative to the convex-upsampling variant.

F RLBench Tasks

The RLBench evaluation uses 18 multitask, multi-variation tasks, while the accompanying task descriptions specify MemoryBench-style sequential tasks that require recalling earlier states or actions.

  • RLBench Task Suite: 18 RLBench tasks cover 249 variations in object placement, color, size, category, count, and shape.The setup follows prior PerAct, RVT, and RVT-2 multi-task, multi-variation experiments.
  • RLBench Task Suite: Table 10 reports each RLBench task’s language template, average extracted keyframes, task variations, and variation type.
  • Memory Tasks: The reopen-drawer task requires remembering which drawer slot was initially opened, pressing a button between actions, and reopening that drawer.Success is defined by reopening the initially opened drawer.
  • Memory Tasks: The put-block-back task moves a block to the centre, presses a button, and returns the block to its initial position.Success is defined by restoring the block to its initial pose.
  • Memory Tasks: The rearrange-block task moves one block to an empty patch, presses a button, and moves the alternative block to the centre.Success is defined by moving the alternative block to the centre.
  • Task Revision: The updated reopen-drawer task adds a closing waypoint to avoid gripper collisions and retrains all policies on the revised task.The update also standardizes the memory-window size across the three tasks.

I Real-world Experiments

The real-world evaluation uses a Franka Panda setup with depth sensing and tests lamp control, ordered button pressing, block stacking, and button-location recall.

  • Setup: The real-world setup uses a Franka Panda robot, Robotiq gripper, and Intel RealSense D455 depth sensor.Figure 5 identifies the robot, gripper, and exocentric depth sensor used to capture the scene.
  • Tasks: The lamp task requires rotating a knob precisely until the lamp turns on.Success is defined by turning on the lamp through knob rotation.
  • Tasks: The button task requires pressing the red button before the blue button while leaving a third button untouched.Success depends on the specified red-then-blue order and avoiding the third button.
  • Tasks: The stacking task places one specified block on another specified block, requiring precise placement and correct language-based identification.Success requires stacking the designated block on the correct target block.
  • Tasks: The memory task requires pressing the button nearest the blue block, removing the block, and pressing the same button again.Pressing the other button at any point causes failure, so the robot must retain the button’s location after removal.
Loading 2501.18564v4…