PaperlayerView exact revision on arXiv

Source-linked AI summary

CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation

Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, Xiaofan Wang, Bei Liu, Jianlong Fu, Jianmin Bao, Dong Chen, Yuanchun Shi, Jiaolong Yang, Baining Guo

arXiv:2411.19650v1cs.ROcs.AIcs.CLcs.CVcs.LG

TL;DR

Existing VLAs offer generalization but their action-prediction strategies can limit task performance. CogACT separates cognition from action with a VLM-conditioned diffusion-transformer action module, and reports stronger performance and generalization across robot embodiments and evaluation settings.

  • Problem

    Existing VLA adaptations use simple action quantization or regression heads that limit action precision or overlook the probabilistic and multimodal nature of robot actions.

  • Method

    CogACT uses VLM cognitive features to condition a specialized diffusion-transformer action module within a componentized VLA architecture.

  • Results

    CogACT significantly surpasses previous VLAs in task performance while showing adaptation to new robots and tasks and generalization to unseen objects and backgrounds.

  • Takeaways & Limitations

    The findings support componentized VLA design with a large VLM for cognition and a dedicated DiT module for precise sequential action prediction.

  • Takeaways & Limitations

    The study considers a gripper action space with 7 degrees of freedom.

Abstract

from arXiv · show

The advancement of large Vision-Language-Action (VLA) models has significantly improved robotic manipulation in terms of language-guided task execution and generalization to unseen scenarios. While existing VLAs adapted from pretrained large Vision-Language-Models (VLM) have demonstrated promising generalizability, their task performance is still unsatisfactory as indicated by the low tasks success rates in different environments. In this paper, we present a new advanced VLA architecture derived from VLM. Unlike previous works that directly repurpose VLM for action prediction by simple action quantization, we propose a omponentized VLA architecture that has a specialized action module conditioned on VLM output. We systematically study the design of the action module and demonstrates the strong performance enhancement with diffusion action transformers for action sequence modeling, as well as their favorable scaling behaviors. We also conduct comprehensive experiments and ablation studies to evaluate the efficacy of our models with varied designs. The evaluation on 5 robot embodiments in simulation and real work shows that our model not only significantly surpasses existing VLAs in task performance and but also exhibits remarkable adaptation to new robots and generalization to unseen objects and backgrounds. It exceeds the average success rates of OpenVLA which has similar model size (7B) with ours by over 35% in simulated evaluation and 55% in real robot experiments. It also outperforms the large RT-2-X model (55B) by 18% absolute success rates in simulation. Code and models can be found on our project page (https://cogact.github.io/).

1. Introduction

CogACT addresses limitations in existing VLA action prediction by separating cognition from action and using a specialized diffusion-based action module. Experiments and scaling studies report improved performance, adaptation, and generalization.

  • Motivation: Existing VLAs can generalize but often achieve unsatisfactory task performance because simple action quantization limits precision and regression heads overlook action multimodality.These approaches also mismatch the continuous nature of robot actions.
  • Approach: CogACT uses VLM-derived cognitive information to condition a specialized action module based on diffusion transformers.The architecture targets continuous, multimodal, temporally correlated, and high-precision action signals.
  • Findings: Sequential diffusion-transformer action modeling significantly outperforms single-step prediction, while adding several hundred million action-module parameters to a 7B VLM yields sizable gains.The study reports favorable scaling behavior for the specialized action module.
  • Findings: An Adaptive Action Ensemble fuses past action predictions adaptively and brings notable performance improvement.The contribution is evaluated alongside the model and action-module studies.
  • Evaluation: Evaluation across simulation and real-robot benchmarks shows strong performance, quick adaptation to new robots and tasks, and generalization to unseen objects and backgrounds.The paper trains on Open X-Embodiment and evaluates across multiple settings.

2. Related Works

Related VLA work adapts pretrained vision-language models for action generation, while large action models and diffusion policies explore more specialized action modeling. These approaches differ in how visual and language information are encoded and integrated with action prediction.

  • Vision-Language-Action Models: RoboFlamingo adds an action-prediction head to OpenFlamingo, whereas RT-2 and OpenVLA tokenize robot actions into discrete language-like tokens.These methods extend or adapt pretrained VLMs for robotic action generation.
  • Vision-Language-Action Models: Token-based VLA methods benefit from VLM capabilities but do not fully account for actions being continuous and temporal rather than language-like.This distinction motivates specialized action modeling.
  • Large Action Models: Concurrent large action models train diffusion transformers with separately pretrained and frozen vision and language encoders to integrate inputs and predict actions.Reported action-model sizes include 221M and 1B parameters.
  • Diffusion-Based Robot Policies: Diffusion-based robot policies are designed to capture multimodal action distributions and multiple feasible trajectories for a task.This line of work motivates diffusion approaches for robotic action generation.

3. Method

CogACT componentizes VLA processing into vision, language, diffusion action, and adaptive ensemble modules. Visual and language information produces a cognition feature that conditions multi-step action generation, while adaptive fusion combines compatible current and historical predictions.

  • Action space: The action representation includes 3D translation offsets, three rotation changes, and a binary gripper state for a 7-DoF gripper.The action sequence is defined over the current and future time steps.
  • Overall architecture: CogACT decomposes the model into vision, language, and action modules that transform observations and instructions into predicted robot actions.The vision module produces visual tokens, the language module integrates them with instructions, and the action module generates actions.
  • Vision module: The vision module combines DINOv2 and SigLIP features into 256 visual perceptual tokens by default.The feature maps are concatenated, linearly projected, and serialized into visual tokens.
  • Language module: The language module combines linguistic tokens, visual tokens, and a learnable cognition token to produce a cognition feature conditioning action generation.The cognition feature encodes integrated information determining the action for the current task.
  • Diffusion action module: The diffusion action module denoises noisy action sequences conditioned on cognition features and predicts current plus future actions.It uses a diffusion transformer to model continuous, multimodal, temporally correlated actions; N = 15 future actions are predicted by default.
  • Adaptive action ensemble: Adaptive ensemble weighting uses cosine similarity to emphasize historical predictions resembling the current prediction, improving success rates with minimal inference cost.The method addresses jerky trajectories and incompatible aggregation of different action modes; cached past predictions limit additional cost.

4. Experiment

Experiments evaluate CogACT in simulation and on two real-robot platforms, including standard task performance, cross-robot transfer, and generalization to unseen visual conditions. Ablations examine action-module scaling, future-step prediction, and adaptive temporal ensembling.

  • Experimental Setup: 22.5 million frames from the Open X-Embodiment subset support training across 22 robot embodiments, with end-to-end training of vision, language, and action modules.The broader OXE dataset contains over 1 million trajectories from 60 datasets.
  • Simulated Evaluation: CogACT attains 74.8% success in Visual Matching and 61.3% in Variant Aggregation on the Google robot, the highest average rates in both settings.The model also exceeds RT-1 by 22.4% and 17.6% average success-rate margins in the two settings.
  • Simulated Evaluation: 51.3% average success rate on the WidowX robot is the highest result in SIMPLER Visual Matching among the compared models.Each task was repeated five times for these tests.
  • Real-World Evaluation: 59.1% is CogACT’s success-rate improvement over OpenVLA in Realman experiments across the evaluated real-world tasks.All compared models were pretrained on OXE and fine-tuned using the collected demonstrations.
  • Generalization Evaluation: CogACT generalizes to unseen tables, distractors, colors, shapes, and categories, with particularly strong performance on unseen colors and shapes.The evaluation modifies table appearance, introduces distractor objects, and tests new object-color, shape, and category combinations.
  • Real-World Evaluation: Real-world evaluation on Realman and Franka robots consistently supports CogACT’s generalization across different robotic platforms.The Realman evaluation covers Pick, Stack, and Place tasks; Franka results use 11 trials per task.
  • Ablation Study: 64.8% average success rate is achieved by DiT-Large, while transformer success rates increase approximately linearly with the logarithm of action-module size.At equal parameter counts, transformers outperform MLPs, consistent with stronger sequence-modeling capability.
  • Ablation Study: Predicting 15 future action steps produces the highest performance among the tested horizons of 0, 3, 15, and 31 steps.The study evaluates multi-step action prediction during training.

5. Conclusion

CogACT separates cognition from action through a specialized diffusion-transformer action module conditioned on a large VLM. This componentized design improves task performance and generalization, with favorable scaling of the action module.

  • CogACT uses a large VLM as the cognitive foundation and a dedicated DiT module for precise, sequential action prediction.The architecture separates cognitive and action capabilities rather than directly repurposing the VLM for action prediction.
  • The specialized action module addresses the continuous, multimodal, and temporally correlated nature of robot actions.
  • Modest parameter increases in the action module yield significant performance gains.
  • Extensive experiments show that CogACT surpasses existing VLAs in task performance and generalizes to unseen objects and backgrounds.

A.1. Pretraining Data

CogACT builds on pretrained vision-language components and pretrains its VLA using selected datasets from Open X-Embodiment. The training setup combines large-scale visual, language, and robotic data sources.

  • The pretrained Prismatic VLM uses DINOv2 and SigLIP vision modules alongside a LLaMA-2 language backbone.DINOv2 and SigLIP were trained on 1.2 billion images and 40 billion image-text pairs, while LLaMA-2 was trained on 2 trillion language tokens.
  • CogACT pretrains on 25 VLA datasets from Open X-Embodiment containing single-arm end-effector control and at least one third-person camera view.Language Table and Droid are excluded because of their significant distribution disparities with the other training data.
  • The data mixture follows the strategies used by Octo and OpenVLA, with restrictions on dataset selection and composition.

A.2. Finetuning Data for Real Robot Experiments

Real-robot fine-tuning uses Realman and Franka embodiments with distinct hardware, sensing, coordinate systems, and demonstration rates. The data is converted into a common training format with image and action preprocessing.

  • Robot setups: The Realman setup uses one 7-DoF arm with a 1-DoF gripper and an Intel RealSense RGB camera, while the Franka setup uses a 7-DoF arm and Kinect DK camera.Realman experiments use only the left arm; Franka actions remain in the robot coordinate system.
  • Data collection: Demonstrations are collected by Meta Quest 2 touch-controller teleoperation, with translation and rotation mapped to gripper motion.Video is recorded at 30 Hz for Realman and 5∼6 Hz for Franka.
  • Preprocessing: Both fine-tuning datasets follow the Open X-Embodiment format, using 224×224 images and relative translation and rotation actions.Rotation changes are represented through rotation matrices converted to Euler angles for training.
  • Preprocessing: Realman data is randomly subsampled to 5 Hz, whereas Franka data is used at its recorded rate.
  • Preprocessing: Images receive crop, brightness, contrast, saturation, and hue augmentation, while actions are normalized to [−1, 1].Stationary actions pad predictions beyond demonstration length; Gaussian normalization did not improve performance.

B.1. Simulated Evaluation

CogACT is evaluated in simulated manipulation environments using SIMPLER, which supports reproducible assessment of policies trained on real data. The protocol covers multiple robot setups, task variants, repeated trials, and adaptive temporal action fusion.

  • Evaluation framework: SIMPLER evaluates policies trained on real data in simulation to provide an easily reproducible and fair manipulation assessment.The framework addresses variation in hardware, environments, and tasks while reducing the resources required by standardized real-world evaluations.
  • Tasks: The Google robot evaluation includes pick, move, drawer, and placement tasks, with both Visual Matching and Variant Aggregations evaluations.
  • Tasks: The WidowX evaluation includes spoon, carrot, block-stacking, and eggplant-placement tasks.
  • Protocol: WidowX trials are repeated five times with different random seeds because the original protocol contains only 24 trials per task.The repetitions are intended to enhance statistical significance.
  • Inference: Simulated inference uses DDIM sampling with 10 steps and classifier-free guidance coefficient 1.5.
  • Action fusion: Adaptive action ensemble uses historical observations and predictions, with K selected so K × std remains constant across datasets.The constant C is set to 0.2, accounting for differences in control frequency and robot speed.

B.2. Real-World Evaluation

The real-world evaluation defines robot tasks across Realman and Franka setups and illustrates executions on both embodiments.

  • Realman tasks: Realman evaluation defines Pick, Stack, and Place tasks in seen and unseen-table environments.The corresponding setups are shown in Figures III and IV.
  • Franka tasks: Franka evaluation covers closing and opening an oven door, picking up a green brush, and picking up a bowl containing food.These task setups are presented in Figure IX.
  • Implementation: All compared models were fully finetuned with PyTorch FSDP using a batch size of 256 on 16 NVIDIA A100 GPUs.The method used one checkpoint at 10K finetuning steps, requiring 7.5 hours.
  • Visualization: Figures VII and VIII show the default model executing Realman tasks, while Figure IX shows executions on the Franka robot.

C.1. Action Model Architectures

The action-model study compares architectures and guidance settings alongside visual demonstrations across multiple robot embodiments. Transformer-based action modules and classifier-free guidance are highlighted in the supplied findings.

  • Action architectures: Larger MLP and transformer action models improve success rates, while transformers outperform MLPs at matched parameter counts.The comparison attributes this advantage likely to attention-based sequence modeling.
  • Robot demonstrations: The supplied figures visualize tasks driven by the model on Google, WidowX, Realman, and Franka robots.The Realman examples include seen tasks and unseen-table-with-unseen-distractors tasks.
  • Guidance: Classifier-free guidance is applied during inference with a default scale of 1.5 because experiments show improved success rates.Different guidance scales are compared in Table IV.
Loading 2411.19650v1…