PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulation

Jiaming Liu, Mengzhen Liu, Zhenyu Wang, Pengju An, Xiaoqi Li, Kaichen Zhou, Senqiao Yang, Renrui Zhang, Yandong Guo, Shanghang Zhang

arXiv:2406.04339v2cs.CV

TL;DR

Robot VLA models face challenges in complex reasoning and computationally costly fine-tuning and inference. RoboMamba combines a vision encoder with Mamba and adds a lightweight policy head for pose prediction. It reports strong reasoning and pose-prediction results, with inference 3 times faster than previous robotic VLA models.

  • Problem

    Existing robot MLLM policies have insufficient reasoning for complex tasks and high fine-tuning and action-generation costs from attention-based language models.

    Source: §1 · p. 2

  • Method

    RoboMamba aligns a vision encoder with Mamba for robotic reasoning, then fine-tunes a simple policy head for end-effector pose prediction.

    Source: §1 · pp. 2–3

  • Results

    RoboMamba excels in reasoning on general and robotic benchmarks and pose prediction in simulation and real-world experiments, with inference 3 times faster than previous robotic VLA models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    A policy head containing 0.1% of model parameters can add manipulation skills while preserving RoboMamba's inherent reasoning capabilities.

    Source: §1 · p. 2, §5 · p. 10

Abstract

from arXiv · show

A fundamental objective in robot manipulation is to enable models to comprehend visual scenes and execute actions. Although existing Vision-Language-Action (VLA) models for robots can handle a range of basic tasks, they still face challenges in two areas: (1) insufficient reasoning ability to tackle complex tasks, and (2) high computational costs for VLA model fine-tuning and inference. The recently proposed state space model (SSM) known as Mamba demonstrates promising capabilities in non-trivial sequence modeling with linear inference complexity. Inspired by this, we introduce RoboMamba, an end-to-end robotic VLA model that leverages Mamba to deliver both robotic reasoning and action capabilities, while maintaining efficient fine-tuning and inference. Specifically, we first integrate the vision encoder with Mamba, aligning visual tokens with language embedding through co-training, empowering our model with visual common sense and robotic-related reasoning. To further equip RoboMamba with SE(3) pose prediction abilities, we explore an efficient fine-tuning strategy with a simple policy head. We find that once RoboMamba possesses sufficient reasoning capability, it can acquire manipulation skills with minimal fine-tuning parameters (0.1\% of the model) and time. In experiments, RoboMamba demonstrates outstanding reasoning capabilities on general and robotic evaluation benchmarks. Meanwhile, our model showcases impressive pose prediction results in both simulation and real-world experiments, achieving inference speeds 3 times faster than existing VLA models. Our project web page: https://sites.google.com/view/robomamba-web

1 Introduction

Robot manipulation models face gaps in complex reasoning and computationally costly fine-tuning and inference. RoboMamba combines vision with Mamba for reasoning, then adds a lightweight policy head for pose prediction; experiments report strong reasoning and pose-prediction results.

Motivation

  • Existing MLLM-based policies struggle with complex robotic reasoning and incur high fine-tuning and action-generation costs from attention-based language models.The introduction frames the research question as whether a VLA model can combine strong reasoning with cost-effective manipulation skills.Source: §1 · p. 2

Contributions

  • RoboMamba combines a vision encoder with Mamba for visual common sense and robotic reasoning, then uses an efficient policy-head fine-tuning strategy for pose prediction.The policy head contains 0.1% of model parameters, 10 times smaller than existing robotic VLA approaches.Source: §1 · pp. 2–3

Experiments

  • RoboMamba excels in reasoning on general and robotic benchmarks and shows pose-prediction results in simulation and real-world experiments.The reported results include 42.8 BLEU-4 on RoboVQA, SOTA manipulation performance in SAPIEN, and inference 3 times faster than previous robotic VLA models.Source: §1 · p. 3

2 Related work

Prior work spans state space models, multimodal language models, and robot-manipulation systems. RoboMamba positions itself as an efficient VLA model that combines robotic reasoning with low-level pose prediction.

State Space Models (SSMs)

  • Mamba uses input-dependent state-space matrices for selective sequence modeling, while prior work applies SSMs to vision, video, restoration, pan-sharpening, and diffusion.The paper identifies its work as the first attempt to introduce Mamba to robotics for efficient action capabilities.Source: §2 · p. 3

Multimodal Large Language Models

  • Prior MLLMs expand visual or 3D understanding, while smaller models and Mamba-based Cobra target reasoning with lower computational costs.RoboMamba instead targets an efficient robotic VLA with common-sense understanding and manipulation capabilities.Source: §2 · p. 3

Robot Manipulation

  • Robot-manipulation methods range from state-based learning and geometric predictions to MLLM-based planning and pose-prediction systems.RoboMamba's stated distinction is combining robotic-related reasoning with low-level pose prediction rather than fine-tuning a pre-trained MLLM.Source: §2 · pp. 3–4

3 RoboMamba

RoboMamba maps visual and language inputs to reasoning responses and end-effector actions. Its design uses a Mamba-based multimodal architecture, staged reasoning training, and frozen-model manipulation fine-tuning with small policy heads.

3 RoboMamba

  • The section roadmap covers the task and language-model preliminaries, architecture, reasoning training, and manipulation fine-tuning.The final stage is presented as equipping the model with pose-prediction skills using minimal parameters and time.Source: §3 · p. 4

3.1 Preliminaries

  • For manipulation, RoboMamba predicts an end-effector pose with 6-DoF position and rotation, adding gripper status for 7-DoF grasp control.The model also generates language answers from an image and question, with answers potentially decomposed into subtasks.Source: §3.1 · p. 4

3.2 RoboMamba architecture

  • A CLIP encoder and MLP projection map images into Mamba's language-embedding space, where visual tokens are concatenated with text tokens for sequence modeling.The language output is detokenized into responses; Mamba is selected for context-aware reasoning and linear computational complexity.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

3.3 General and robotic-related training

  • Stage 1 combines image-text alignment pre-training with instruction co-training on general and robotic data to develop common-sense and robotic reasoning.Alignment freezes the vision encoder and Mamba while updating the projection layer; co-training freezes CLIP and fine-tunes the projection layer and Mamba.Source: §3.3 · pp. 5–6

3.4 Robot manipulation fine-tuning

  • Stage 2 freezes RoboMamba and trains simple MLP policy heads for end-effector position and direction, using about 0.1% of total model parameters.In open-loop simulation, the model predicts a 2D contact-pixel position translated to 3D using depth; a 10k-pose dataset supports fine-tuning that takes a few dozen minutes.Source: §3.4 · p. 6

4 Experiment

The experiments evaluate RoboMamba’s reasoning and manipulation in simulation and real-world settings, then examine how model choice and reasoning ability relate to performance. Across these evaluations, the paper reports strong benchmark results alongside efficient manipulation fine-tuning and inference.

4.1 Experiment settings

  • Evaluation combines general and robotic reasoning benchmarks with simulated open-loop manipulation tests and real-world trials.The manipulation dataset contains 10K images across 20 tasks; its test set has 1.1K examples from 20 seen and 10 unseen tasks, while robotic reasoning is evaluated on RoboVQA’s 18K validation set.Source: §4.1 · p. 7

4.2 Reasoning quantitative results

  • RoboMamba is competitive across general VQA and recent MLLM benchmarks, though some results remain below LLaVA1.5 and SPHINX.The authors attribute improved GQA spatial identification to robot data introduced during instruction co-training.Source: §4.2 · pp. 7–8, Table 1 · p. 8
  • On RoboVQA, RoboMamba surpasses LLaMA-AdapterV2 and TinyLLaVA on BLEU-1 through BLEU-4, with inference seven times faster than LLaMA-AdapterV2 and ManipLLM.The baselines were fine-tuned on RoboVQA training data for two epochs using their official instruction-tuning methods.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 Manipulation quantitative results

  • 7.0% improvement on seen tasks and 2.0% on unseen tasks over ManipLLM, with SOTA performance on 14 of 20 seen tasks.The 3.7M-parameter policy head uses 0.1% of model parameters; RoboMamba updates 10 times fewer parameters and has seven times faster inference than the compared MLLM-based methods.Source: §4.3 · p. 8, Table 2

4.4 Ablation study

  • RoboMamba-2.7B outperforms RWKV-3B on common-sense and robotic-related reasoning benchmarks under the same training data and strategy.The authors report that the framework and training strategy can also be adapted to other linear-complexity LLMs.Source: Figure 3 · p. 9, §4.4 · p. 9
  • With the same MLP policy-head fine-tuning strategy, manipulation accuracy tracks reasoning ability, and omitting instruction co-training particularly reduces generalization to unseen objects.The ablation compares RoboMamba-2.7B, OpenFlamingo, LLaMA-AdapterV2, RoboMamba-1.4B, and RoboMamba-2.7B without co-training.Source: Figure 3 · p. 9, §4.4 · p. 9

4.5 Real-world experiments

  • Real-world visualizations cover planning, affordance, past and future prediction, and low-level pose prediction; RoboMamba is described as more accurate than LLaMA-AdapterV2 on task planning.For pose visualization, a red dot marks the contact point and the end-effector depicts rotation; the paper also visualizes failure cases.Source: Figure 4 · p. 10, §4.5 · p. 10

5 Conclusion and future plan

RoboMamba combines a vision encoder with linear-complexity Mamba and adds manipulation skills through a small policy head. The authors report strong reasoning and pose-prediction results, while noting limits on complex reasoning relative to larger MLLMs.

Conclusion

  • RoboMamba combines a vision encoder with linear-complexity Mamba, with visual common-sense and robotic reasoning abilities.The conclusion describes imparting manipulation skills by fine-tuning a simple policy head comprising 0.1% of the model in a few dozen minutes.Source: §5 · p. 10
  • The paper reports strong general and robotic-related reasoning benchmark results and pose-prediction results.The authors characterize the policy-head approach as equipping the VLA model with manipulation abilities without compromising its inherent reasoning capabilities.Source: §5 · p. 10

Limitations and future plan

  • Reliance on a 2.7B LLM limits some complex reasoning tasks relative to MLLMs built on 7B or 13B LLMs.Future directions include adapting the framework to more advanced linear-complexity LLMs and constructing a 4D Robot VLA model using 3D point-cloud and temporal data.Source: §5 · p. 10

A Appendix

The appendix points readers to supplementary details on datasets, ablations, qualitative results, and evaluation choices.

Appendix B

  • Appendix B gives additional descriptions of the alignment pre-training, instruction co-training, and robot manipulation fine-tuning datasets.It includes a more detailed description of the training dataset.Source: A Appendix · p. 17

Appendix C

  • Appendix C explores image encoders, training datasets, and policy-head designs through additional ablations.Source: A Appendix · p. 17

Appendix D

  • Appendix D presents additional qualitative results across robot-related downstream tasks.Source: A Appendix · p. 17

Appendix E

  • Appendix E explains metric selection and prompt usage during testing.Source: A Appendix · p. 17

Stage 1.1: Alignment pre-training dataset

The appendix describes the datasets used for instruction co-training and simulator-based manipulation fine-tuning, including the training and testing variations used to assess generalization.

Alignment pre-training dataset

  • LLaVA-LCS 558K is a concept-balanced subset of LAION/CC/SBU with captions paired with BLIP synthetic captions for reference.It is used as the cross-modal alignment dataset.Source: Stage 1.1: Alignment pre-training… · p. 17

Instruction co-training datasets

  • For co-training, the study randomly selects 300K image-text paired instruction samples from RoboVQA, which contains 5,246 long-horizon and 92,948 medium-horizon robotic-task episodes.Source: Stage 1.1: Alignment pre-training… · p. 17

Category task design

  • Category-specific tasks include opening refrigerator doors or control panels, opening bottle caps, and rotating box lids.The appendix points to the SAPIEN task-design catalog for further examples.Source: Stage 1.1: Alignment pre-training… · pp. 17–18

Simulator data collection

  • Simulator collection uses a Franka Panda with suction gripper and yields 10K images across 20 categories for training and 1.1K images for testing.Successful interactions are added to the training set; test images include categories seen during training and unseen categories.Source: Stage 1.1: Alignment pre-training… · p. 18

Asset variation

  • Asset variation separates training and test shapes within seen categories and reserves 10 categories and 274 shapes exclusively for unseen-category testing.The 20 seen categories use 1,037 training shapes and 489 test shapes.Source: Stage 1.1: Alignment pre-training… · p. 18

State variation

  • State variation randomizes camera azimuth and altitude and initializes articulated parts from fully closed to half-open in both training and testing.The RGB-D camera is placed 4.5–5.5 units from the object, on its upper hemisphere and facing its center.Source: Stage 1.1: Alignment pre-training… · p. 18

C Additional ablation study

Additional ablations examine image encoders, training datasets, and policy-head designs. The reported findings favor robotic instruction data for robotic reasoning, while encoder choice and head size show limited influence in the tested settings.

Impact of different image encoders

  • Table 4 reports that encoder choice and input resolution do not significantly affect reasoning, while XCiT without cross-modal alignment makes mapping image tokens to language embeddings challenging.The authors identify projection of image tokens into language embeddings at linear computational complexity as future work.Source: Table 4 · p. 18, C Additional ablation study · p. 18

Impact of training datasets

  • Table 5 comparisons indicate that robotic instruction data enhances robotic-specific reasoning, while more advanced general instruction datasets yield no significant gains.The authors suggest the latter may reflect the 2.7B LLM’s model capacity; their generated robotic-scene descriptions also show improved understanding of geometric relationships with robotic instruction data.Source: C Additional ablation study · pp. 18–19, Table 5 · p. 19

Impact of policy-head designs

  • Table 6 shows similar manipulation accuracy across three policy-head designs, supporting low-cost pose-skill learning regardless of head design once reasoning is sufficient.The designs are one MLP head, separate position and direction MLP heads, and two such heads preceded by SSM blocks.Source: Table 6 · p. 19, C Additional ablation study · p. 19

D Additional real-world experiments

Real-world experiments use a modified Franka Emika arm to demonstrate reasoning across several downstream tasks. The authors also report reasoning errors on some complex tasks.

Real-world setup

  • Real-world demonstrations use a Franka Emika arm whose finger gripper is modified with double-sided tape to provide suction.The demonstrations are included in the supplementary video.Source: D Additional real-world experiments · p. 19

Demonstrated tasks

  • The demonstrations visualize long-horizon planning, discriminative and generative affordance, past description, and future prediction.The passage also refers to task-planning and low-level pose-prediction visualizations in Figure 5.Source: D Additional real-world experiments · p. 19

Reasoning failures

E Reasoning evaluation bencharks

The appendix describes the reasoning benchmarks and visualizations used to assess general vision understanding, robotic reasoning, and multimodal skills. The figures also cover downstream robotic tasks and reasoning failures.

Vision-questioning benchmarks

  • VQAv2 and OKVQA assess basic visual question answering, while POPE and VizWiz test responses to visual illusions or ambiguities.The appendix frames these abilities as relevant to embodied AI and avoiding significant errors in robotic applications.Source: E Reasoning evaluation bencharks · p. 19

Real-world task visualizations

  • Figure 5 visualizes task planning, long-horizon planning, affordance, past and future prediction, and low-level pose prediction across real-world robotic tasks.The affordance examples include discriminative and generative affordance.Source: Figure 5 · p. 20

Failure-case visualizations

  • Figure 6 presents reasoning failure cases and replaces the virtual end-effector with a physical Franka Panda Robot in the selected demonstration.The figure description also notes a red dot in the image.Source: Figure 6 · p. 20

Reasoning benchmarks

  • GQA tests identification and localization of important objects, RobotVQA tests action planning from visual and textual inputs, and MM-Vet, MME, and MMB cover multimodal task abilities.The listed MM-Vet, MME, and MMB indicators include recognition, spatial awareness, OCR, and math.Source: E Reasoning evaluation bencharks · p. 20

Related papers

Loading 2406.04339v2…