PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models

Dongyoung Kim, Sumin Park, Woomin Song, Seungku Kim, Taeyoung Kim, Huiwon Jang, Jinwoo Shin, Jaehyung Kim, Younggyo Seo

arXiv:2603.21341v1cs.AI

TL;DR

Existing language-supervised embodied-reasoning methods do not reliably improve VLA performance, motivating direct alignment between MLLM reasoning and low-level actions. RoboAlign first enables action-token generation through SFT and then refines reasoning with RL using action accuracy. It improves SFT baselines by 17.5% on LIBERO, 18.9% on CALVIN, and 106.6% in real-world environments using less than 1% additional data.

  • Problem

    Embodied-reasoning improvements in MLLMs correlate inconsistently with VLA performance because language supervision does not directly optimize low-level action generation.

  • Method

    RoboAlign uses SFT for zero-shot low-level action-token generation, then applies GRPO to refine reasoning with an action-accuracy reward.

  • Results

    17.5%, 18.9%, and 106.6% relative improvements over SFT-only baselines were achieved on LIBERO, CALVIN, and real-world environments, respectively.

  • Takeaways & Limitations

    ROBOALIGN consistently improves embodied reasoning and VLA performance across robotic environments, including short- and long-horizon tasks.

  • Takeaways & Limitations

    Aligning an MLLM with unsafe action trajectories may bias resulting VLAs toward unsafe actions, requiring proactive training-time prevention.

Abstract

from arXiv · show

Improving embodied reasoning in multimodal-large-language models (MLLMs) is essential for building vision-language-action models (VLAs) on top of them to readily translate multimodal understanding into low-level actions. Accordingly, recent work has explored enhancing embodied reasoning in MLLMs through supervision of vision-question-answering type. However, these approaches have been reported to result in unstable VLA performance, often yielding only marginal or even negative gains. In this paper, we propose a more systematic MLLM training framework RoboAlign that reliably improves VLA performance. Our key idea is to sample action tokens via zero-shot natural language reasoning and refines this reasoning using reinforcement learning (RL) to improve action accuracy. As a result, RoboAlign bridges the modality gap between language and low-level actions in MLLMs, and facilitate knowledge transfer from MLLM to VLA. To validate the effectiveness of RoboAlign, we train VLAs by adding a diffusion-based action head on top of an MLLM backbone and evaluate them on major robotics benchmarks. Remarkably, by performing RL-based alignment after SFT using less than 1\% of the data, RoboAlign achieves performance improvements of 17.5\%, 18.9\%, and 106.6\% over SFT baselines on LIBERO, CALVIN, and real-world environments, respectively.

1. Introduction

VLAs do not reliably benefit from language-supervised embodied reasoning because language reasoning does not directly optimize low-level actions. RoboAlign addresses this modality gap by aligning reasoning with action accuracy through SFT followed by RL.

  • VLAs remain limited by MLLMs’ difficulties with spatial and temporal reasoning needed for action generation.
  • Embodied-reasoning improvements correlate inconsistently with VLA performance and can even degrade it across tasks.
  • ROBOALIGN generates low-level action tokens through zero-shot reasoning, then uses GRPO to refine reasoning with an action-accuracy reward.
  • ROBOALIGN evaluates VLAs by attaching a diffusion-based action head to an MLLM backbone across simulated and real-world robotic benchmarks.
  • 17.5%, 18.9%, and 106.6% relative improvements over SFT-only baselines were achieved on LIBERO, CALVIN, and real-world environments, respectively.The RL alignment stage used less than 1% additional data after SFT.
  • ROBOALIGN also achieves state-of-the-art embodied reasoning while preserving general image-understanding performance.

2. Related Work

Prior work improves MLLM embodied reasoning through planning, spatial grounding, and reinforcement-learning-based reasoning methods. However, much of the supervision remains indirect and does not directly optimize low-level action generation.

  • MLLMs have been used for high-level action planning and predefined atomic skills in robot control.
  • VQA-based methods enhance spatial and 3D scene understanding for embodied applications.
  • Bounding-box and point-prediction approaches provide indirect supervision rather than directly optimizing low-level action generation.
  • Chain-of-Thought prompting improves answer quality across zero-shot, few-shot, and supervised fine-tuning settings for LLMs and MLLMs.

3. Preliminaries

RoboAlign integrates low-level actions into MLLMs through FAST tokenization and trains explicit reasoning with GRPO. FAST compresses action chunks into token sequences, while GRPO optimizes sampled outputs using normalized rewards and a reference-policy constraint.

  • FAST represents each action as end-effector position, orientation, and gripper state across a horizon of H timesteps.
  • FAST transforms action chunks into frequency-domain DCT coefficients, then quantizes and flattens them into a sequence.
  • GRPO trains the model to produce intermediate thoughts before a final answer while optimizing format correctness and answer accuracy.
  • GRPO uses the importance-sampling ratio between current and old policies to weight sampled outputs.
  • GRPO maximizes normalized advantages while applying a KL penalty against a reference policy.

4. ROBOALIGN: Align Embodied Reasoning with Low-level Actions

ROBOALIGN aligns MLLM reasoning with low-level FAST actions through a two-stage process: robotics-aware SFT followed by GRPO-based reinforcement learning. The framework builds action-generation capability during SFT, then optimizes reasoning and token accuracy for VLA adaptation.

  • Framework overview: ROBOALIGN uses robotics-aware SFT followed by GRPO reinforcement learning to align MLLM reasoning with low-level FAST actions.Stage 1 establishes action-token generation, while Stage 2 refines the reasoning process and action accuracy.
  • Stage 1: SFT: The SFT stage equips the MLLM to generate FAST action tokens while preserving general vision-language capabilities and enhancing embodied reasoning.Its data mixture combines embodied and general SFT datasets, custom robotic VQA, zero-shot reasoning data, and robotic FAST-token data.
  • Stage 1: SFT: ROBOALIGN’s robotic VQA pipeline generates spatial-temporal question-answer, captioning, and grounding pairs from robot images and metadata.The metadata includes bounding boxes, end-effector states, and high- and low-level actions.
  • Stage 1: SFT: Zero-shot reasoning data is incorporated during SFT to transfer step-by-step reasoning ability into the action-generation process.The reasoning data is distilled from trajectories sampled from a GRPO-trained reasoning model.
  • Stage 1: SFT: The FAST-token dataset extends the vocabulary with action markers and 2K FAST tokens, then pairs robot images with ground-truth FAST-token sequences in QA format.This data mixture provides the foundation for subsequent RL training.
  • Stage 2: RL: Stage 2 prompts explicit reasoning before FAST-token generation and uses a reward combining format adherence with FAST-token accuracy, optimized through GRPO.The accuracy component measures prefix similarity between generated and target action-token sequences.

5. Experiment

The experiments evaluate whether ROBOALIGN improves VLA performance across simulation, real-robot, alternative alignment strategies, model backbones, and MLLM benchmarks. Across these settings, ROBOALIGN provides the strongest gains, especially on long-horizon tasks, while RL-based low-level action alignment sharpens representations and preserves MLLM capabilities.

  • 5.1. Experimental Setup: The evaluation uses a unified VLA conversion pipeline with a diffusion-based action head and frozen MLLM backbone across LIBERO, CALVIN, and real-robot benchmarks.The study also compares alternative MLLM training methods, alignment strategies, and model architectures.
  • 5.2. Main Results: ROBOALIGN achieves the highest simulated performance, with most gains arising from RL despite using less than 1% of the SFT data.The SFT stage uses 2.28M samples, whereas RL uses 12.8K samples.
  • 5.2. Main Results: 70% LIBERO Long success and 22.2% CALVIN length-5 success show ROBOALIGN’s strongest advantage on long-horizon tasks.CALVIN length-5 success rises from 18.1% to 22.2%, while other methods decline; LIBERO Long reaches 70% versus approximately 2% gains from other methods.
  • 5.2. Main Results: Language-only reasoning improves simpler tasks but transfers poorly to complex long-horizon tasks, while action-only training can reduce zero-shot generalization.The paper attributes the latter pattern to forgetting of general MLLM capabilities during FAST token training.
  • 5.3. Ablation Study and Analyses: RL alignment increases KNN representation accuracy from 39.06% to 69.79%, while ROBOALIGN also outperforms specialized embodied-reasoning models without degrading general MLLM benchmarks.These analyses support more discriminative state representations and retained MLLM capability.

6. Conclusion

The conclusion presents ROBOALIGN as an MLLM training framework that directly aligns representations with low-level action policies through reinforcement learning. Across robotic environments and MLLM benchmarks, it reports gains in embodied reasoning and VLA performance, including short- and long-horizon tasks.

  • 6. Conclusion: ROBOALIGN directly aligns MLLM representations with low-level action policies using reinforcement learning to improve action prediction accuracy.The framework is tailored to transferring MLLM capabilities into VLAs.
  • 6. Conclusion: ROBOALIGN consistently improves embodied reasoning and VLA performance across diverse environments, including short- and long-horizon tasks.The conclusion contrasts these gains with limited or degraded performance from language-only embodied-reasoning fine-tuning on complex scenarios.

Impact Statement

The impact statement warns that aligning an MLLM with unsafe action trajectories may bias downstream VLAs toward unsafe actions. It emphasizes proactive prevention during training because deployed models’ training data may be difficult to determine precisely.

  • Impact Statement: Aligning MLLMs with unsafe action trajectories may bias VLAs toward generating unsafe actions.The risk concerns the action data used during MLLM alignment.
  • Impact Statement: Because deployed MLLM training data may be difficult to determine precisely, safety risks should be addressed proactively during training.The statement frames prevention as a training-phase responsibility.

A.1. Computing Cost

The appendix reports separate computational requirements for MLLM and VLA training, with RL requiring substantially less time than SFT in the stated setup.

  • A.1. Computing Cost: MLLM training uses 8×H200 GPUs, requiring approximately 30 hours for SFT and 1 hour for reinforcement learning.These are the reported runtimes for the stated training configuration.
  • A.1. Computing Cost: VLA training uses 2×A100 GPUs, with each 10K training steps taking about 1 hour.The reported cost applies to the VLA training setup.

A.2. Implementation Details for VLA Training

The VLA implementation adopts the GR00T-N1.5 architecture with a diffusion policy action expert initialized for training.

  • The implementation follows the GR00T-N1.5 codebase and uses its architecture with an initialized diffusion policy action expert.

A.3. Training Datasets

The experiments combine multiple embodied-reasoning datasets and benchmark ROBOALIGN across multimodal and robotic settings. Results include stronger RL training metrics and more concrete post-training action descriptions.

  • Training Datasets: The SFT data include 100K LLaVA-OneVision samples, 300K RefSpatial samples, 200K RoboPoint samples, 50K EgoPlan-IT samples, and 500K multi-view instruction samples.
  • Performance on MLLM Benchmarks: The evaluation covers general visual question answering, spatial reasoning, and fine-grained robot embodied reasoning benchmarks.
  • Performance on MLLM Benchmarks: ROBOALIGN (SFT) performs on par with specialized embodied-reasoning MLLMs, while RL training further boosts performance across the overall MLLM benchmarks.
  • RL Training Process: During RL training, accuracy reward gradually increases while average response length remains stable, indicating learning without longer responses.
  • RL Training Process: Before RL, responses mainly describe high-level plans; after RL, they provide more concrete immediate actions and situational awareness.
  • RL Training Process: For the orange-cloth task, the ROBOALIGN SFT response predicts action tokens after describing an approach, grasp, and transfer sequence.
Loading 2603.21341v1…