PaperlayerView exact revision on arXiv

Source-linked AI summary

ACoT-VLA: Action Chain-of-Thought for Vision-Language-Action Models

Linqing Zhong, Yi Liu, Yifei Wei, Ziyu Xiong, Maoqing Yao, Si Liu, Guanghui Ren

arXiv:2601.11404v2cs.RO

TL;DR

Existing vision-language-action policies struggle to translate semantic multimodal representations into precise motor commands. ACoT-VLA reasons directly in action space using explicit trajectories and implicit action priors, achieving state-of-the-art performance across real-world and simulation benchmarks.

  • Problem

    Existing generalist robot policies exhibit a semantic-kinematic gap between high-level multimodal representations and precise low-level action execution, motivating action-space guidance.

  • Method

    ACoT-VLA combines Explicit and Implicit Action Reasoners to provide coarse trajectory guidance and latent action priors that condition grounded policy learning.

  • Results

    Across real-world and simulation benchmarks, ACoT-VLA achieves state-of-the-art performance, with EAR and IAR reaching a 98.5% average success rate.

  • Takeaways & Limitations

    The results support action-space deliberation as an effective and grounded approach for generalist robotic policy learning.

  • Takeaways & Limitations

    The reasoning modules add computational cost, and action-chunk representations lack explicit geometric structure for higher-level spatial reasoning.

Abstract

from arXiv · show

Vision-Language-Action models have emerged as essential generalist robot policies for diverse manipulation tasks, conventionally relying on directly translating multimodal inputs into actions via Vision-Language Model embeddings. Recent advancements have introduced explicit intermediary reasoning-such as sub-task prediction (language) or goal image synthesis (vision)-to guide action generation. However, these intermediate reasoning are often indirect and inherently limited in their capacity to convey the full, granular information required for precise action execution. Instead, we posit that the most effective form of reasoning is one that deliberates directly in the action space. We introduce Action Chain-of-Thought (ACoT), a paradigm where the reasoning process itself is formulated as a structured sequence of coarse action intents that guide the final policy. In this paper, we propose ACoT-VLA, a novel architecture that materializes the ACoT paradigm. Specifically, we introduce two complementary components: an Explicit Action Reasoner (EAR) and Implicit Action Reasoner (IAR). The former proposes coarse reference trajectories as explicit action-level reasoning steps, while the latter extracts latent action priors from internal representations of multimodal input, co-forming an ACoT that conditions the downstream action head to enable grounded policy learning. Extensive experiments in real-world and simulation environments demonstrate the superiority of our proposed method. Code is available at: https://github.com/AgibotTech/ACoT-VLA.

1. Introduction

The introduction identifies a semantic-kinematic gap in existing VLA policies, whose language- or vision-based intermediate reasoning provides indirect guidance for precise action generation. ACoT addresses this gap by reasoning directly through structured action-space intents, implemented in ACoT-VLA with complementary explicit and implicit action reasoners.

  • Background: VLA models encode visual and linguistic inputs with a pre-trained VLM before conditioning an action decoder.This paradigm emerged to overcome the generalization limits of task-specific robot policies.
  • Existing reasoning paradigms: Language-based reasoning predicts sub-tasks, while world-model approaches simulate dynamics or synthesize visual guidance to improve action generation.These approaches introduce intermediate reasoning between multimodal inputs and actions, but remain grounded in language or vision.
  • Problem: Existing policies exhibit a semantic-kinematic gap because semantic and visual reasoning provide indirect guidance for precise, low-level executable actions.VLM representations emphasize linguistic understanding, while world-model guidance remains tied to visual representations.
  • Action Chain-of-Thought: ACoT reframes deliberation as a structured chain of coarse action-space intents that provides kinematically coherent guidance for policy learning.The paradigm is introduced to replace abstract linguistic or visual sub-goals with homogeneous action guidance.
  • ACoT-VLA: ACoT-VLA combines an Explicit Action Reasoner for observable reference trajectories with an Implicit Action Reasoner that infers latent action priors through cross-attention.The two complementary reasoners jointly form grounded Action Chain-of-Thought guidance, with effectiveness demonstrated in real-world settings and three simulation benchmarks.

2. Related Works

Related VLA work has progressed from autoregressive discretized action generation to diffusion-based policies, while newer approaches use visual or linguistic intermediaries to guide action prediction. ACoT-VLA instead investigates guidance directly in the action space to reduce perception–action heterogeneity and learn action-relevant priors.

  • Vision-Language-Action Models: VLA models use pretrained vision-language models to predict language-driven robotic action sequences.Early approaches formulate control as autoregressive sequence generation by discretizing continuous actions into bins.
  • Vision-Language-Action Models: Diffusion-based action policies synthesize smooth, high-quality action trajectories as an alternative to autoregressive discretized action generation.These methods are motivated by advances in generative modeling.
  • Visual and Linguistic Intermediaries: Recent methods exploit predictive visual dynamics or sub-goal image forecasting to implicitly or explicitly guide robotic action generation.CoT-VLA uses visual chain-of-thought reasoning by forecasting sub-goal images.
  • Action-Space Guidance: Unlike visual or linguistic intermediaries, ACoT-VLA investigates guidance directly in the action space to mitigate perception–action heterogeneity and learn action-relevant priors.The action-space focus is presented as the paper’s key insight.

3. Methodology

ACoT-VLA shifts policy guidance into the action domain by combining explicit reference trajectories with implicit action cues extracted from multimodal representations. Its Action-Guided Prediction strategy fuses both priors to condition denoised action prediction, with teacher forcing stabilizing training and self-conditioning used at inference.

  • Action-space guidance: ACoT-VLA extends conventional language- and vision-level guidance with action-space guidance, disentangled into explicit reference sequences and implicit contextual motion cues.Explicit guidance provides direct action priors, while implicit guidance arises from contextual signals such as action distributions implied by language.
  • Explicit Action Reasoner: The Explicit Action Reasoner synthesizes kinematically plausible reference action sequences from visual-language context and projects them into explicit action embeddings for policy learning.A VLM supplies contextual key-value caches, while EAR processes noisy action sequences through transformer layers with self- and cross-attention before denoising.
  • Implicit Action Reasoner: The Implicit Action Reasoner extracts complementary action semantics directly from the VLM key-value cache by downsampling layer features, applying cross-attention, and aggregating them across layers.The resulting compact representation captures latent motion cues such as visual affordances and action-related semantics.
  • Action-Guided Prediction: Action-Guided Prediction treats noisy action embeddings as queries that retrieve explicit and implicit priors, then fuses the attended features through concatenation and self-attention.Explicit priors emphasize kinematic cues, whereas implicit priors capture latent action tendencies before the unified representation conditions the action head.
  • Training and inference: Training uses flow-matching MSE objectives for the reasoners and policy, with ground-truth reference trajectories providing teacher-forced explicit embeddings before inference switches to self-conditioned reference generation.Teacher forcing prevents optimization interference, while inference uses autonomously generated reference actions to guide policy prediction.

4. Experiments

Experiments evaluate ACoT-VLA across three simulation benchmarks, systematic ablations, and real-world manipulation tasks. The method consistently improves policy performance, robustness to distribution shifts, and real-world success rates.

  • Experimental Setup: Experiments use official training splits and standard demonstrations for LIBERO, LIBERO-Plus, and VLABench, while real-world training demonstrations come from the authors’ robotic platform.Simulation evaluations cover three benchmarks, followed by ablations and deployment on AgiBot G1.
  • Simulation Results: 1.6% absolute improvement in average performance over π0.5 is achieved on LIBERO, with pronounced gains on the long-horizon LIBERO-Long suite.LIBERO evaluates Spatial, Object, Goal, and Long tracks under the official protocol.
  • Simulation Results: +3.2% robustness on robot initial-state perturbations and +4.2% on language variations are achieved under LIBERO-Plus Zero-Shot Transfer.The method surpasses previous methods in both Zero-Shot Transfer and Supervised Fine-Tuning settings.
  • Simulation Results: 63.5% IS and 47.4% PS are obtained on VLABench, including +12.6% IS and +7.2% PS on the unseen-texture track.IS denotes Intention score and PS denotes Progress score; all models use VLABench’s official training data.
  • Ablation Studies: 98.5% average success rate is achieved when EAR and IAR are combined, while Downsample performs best among IAR interaction strategies.EAR raises the baseline from 96.9% to 98.3%, and the ablations show complementary benefits from explicit and implicit action guidance.
  • Real-World Evaluation: 66.7% average real-world success exceeds π0.5 at 61.0% and π0 at 33.8%, with improvements across AgiBot G1 and AgileX embodiments.Tasks assess contact-rich manipulation, fine-grained object handling, and instruction following under real-world sensing conditions.

5. Conclusion

The paper proposes Action Chain-of-Thought (ACoT), which shifts robotic deliberation into a kinematically grounded action space, and materializes it in the ACoT-VLA framework. Experiments across simulation and real-world benchmarks show state-of-the-art performance and motivate more structured, interpretable, and capable embodied agents.

  • Core paradigm: ACoT addresses the semantic-kinematic gap by proposing deliberation directly in the kinematically grounded space of actions.The paradigm moves reasoning away from abstract language or vision spaces.
  • Framework: ACoT-VLA materializes the paradigm through two synergistic modules for grounded robotic policy learning.The supplied passage identifies the framework’s two-module design but truncates the module names.
  • Empirical results: State-of-the-art performance is demonstrated across multiple simulation and real-world benchmarks.The conclusion attributes these results to the proposed approach.
  • Implications: Shifting reasoning from perception to action provides a more effective and grounded approach while opening research directions for structured, interpretable, and capable embodied agents.The authors frame action-based reasoning as a critical step toward next-generation generalist robots.

A. Dataset Description

The study trains and evaluates on three simulation benchmarks—LIBERO, LIBERO-Plus, and VLABench—and custom real-world demonstrations spanning three manipulation tasks. These datasets vary substantially in scale, recording frequency, trajectory diversity, and physical or visual demands.

  • Dataset Overview: The experiments use LIBERO, LIBERO-Plus, and VLABench as publicly released simulation benchmarks, while real-world data cover Wipe Stain, Pour Water, and Open-set Pick.Dataset statistics include episode counts, frame counts, and recording frequencies.
  • Simulation Benchmarks: 1,693 episodes and 273,465 frames comprise LIBERO, recorded at 10 Hz with relatively uniform trajectory lengths and smooth motion patterns.LIBERO is described as a widely adopted benchmark.
  • Simulation Benchmarks: 14,347 episodes and 2,238,036 frames comprise LIBERO-Plus, recorded at 20 Hz with perturbation-oriented demonstrations and larger motion and viewpoint variations.LIBERO-Plus was introduced as a more challenging and diversified setting amid performance saturation on LIBERO.
  • Simulation Benchmarks: VLABench provides 4,713 training episodes and 528,398 frames at 10 Hz, requiring higher levels of visual and physical understanding from the policy.The reported figures refer to its training set.
  • Real-World Experiment: Real-world demonstrations include 177 Wipe Stain episodes with 356,316 frames, 1,821 Pour Water episodes with 5,062,506 frames, and Open-set Pick subsets on AgiBot G1 and AgileX.Wipe Stain involves dense tool–surface contact and fine-grained force control, while Pour Water is long-horizon and multistage; Open-set Pick has 1,936 AgiBot G1 episodes with 219,824 frames and 962 AgileX episodes with 251,283 frames.

B. Training & Evaluation Details

Training uses benchmark-specific action and state configurations, while real-world tasks use longer-horizon absolute joint control with structured robot observations. Evaluation defines task-specific initializations, success criteria, repeated trials, and operator-assessed success rates.

  • Training Details: LIBERO and LIBERO-Plus use Delta EEF control with a 10-step horizon, no privileged state, batch size 128, and 40K and 100K training steps, respectively.VLABench uses state input, Abs EEF actions, and 60K training steps.
  • Training Details: Real-world tasks use Abs Joint control with a 30-step horizon, structured robot state observations, batch size 128, and 50K, 240K, and 50K training steps.These step counts correspond respectively to “Wipe Stain”, “Pour Water”, and “Open-set Pick”.
  • Evaluation Details: “Wipe Stain” evaluates three sponge poses across four stain locations, with two executions per configuration for 24 trials; success requires grasping the sponge and removing the stain.The task-specific trial definition requires both the grasp and stain removal.
  • Evaluation Details: “Pour Water” evaluates six bottle–glass configurations twice each; success requires lifting, pouring into the cup, and returning the bottle to its coaster, with minor spillage allowed.This protocol totals 12 executions.
  • Evaluation Details: “Open-set Pick” uses ten arrangements containing in-distribution and out-of-distribution objects, evaluates each arm–object pair twice for 40 trials, and requires grasping the instructed object with the correct arm.Evaluations are conducted by trained operators, and success rates equal successful trials divided by total attempts.

C. More Experimental Results

Additional experiments show that ACoT-VLA’s gains arise from action-level reasoning rather than simply increasing model size or denoising steps. EAR improves LIBERO-Plus ablations, matched-budget comparisons favor EAR, and the full approach transfers strongly from simulation to the real world.

  • Ablation Study: Adding EAR raises LIBERO-Plus average success from 75.7% to 83.7% by generating an explicit reference action trajectory.The reference trajectory reduces ambiguity when mapping complex visual or linguistic observations to low-level actions, including camera shifts and background changes.
  • Ablation Study: Adding only IAR improves LIBERO-Plus average success from 75.7% to 80.4%.
  • Effect of Model Scaling & Denoising Budget: Increasing action-head size or denoising steps does not reliably enhance performance, while further denoising produces only negligible fluctuations.
  • Effect of Model Scaling & Denoising Budget: Under fully matched parameterization and denoising budgets, introducing EAR consistently improves performance on both evaluated benchmarks.The comparison pairs are “#1” versus “#3” and “#2” versus “#4”, indicating that the gains originate from the proposed action chain-of-thought.
  • Simulation and Real-world Transfer: 84.2% in simulation and 82.9% in real-world settings, representing absolute improvements of 8.5% and 5.4%, respectively.The approach exhibits minimal degradation during sim-to-real transfer, attributed to consistent kinematically grounded action guidance despite the visual domain gap.

D. Limitations & Future Works

The paper identifies computational overhead and limitations of prevailing action-chunk representations as constraints, while outlining future research directions.

  • The section discusses existing limitations and promising directions for future research.
  • The reasoning modules introduce additional computational cost, potentially challenging deployment on resource-constrained robotic platforms.The authors characterize this cost as relatively modest compared with the performance gains.
  • Another limitation arises from the community’s prevailing action representation, which is implemented as action chunks.
Loading 2601.11404v2…