RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models

Introduction

• Embodied-reasoning gains do not reliably transfer to VLAs: RoboBrain 2.0 had the highest reasoning scores among evaluated MLLMs but the lowest VLA performance.The paper attributes this mismatch to the modality gap between language and low-level actions; Figure 1 reports that embodied-reasoning-specialized MLLMs often fail to improve or degrade LIBERO performance relative to the baseline VLA.Source: §1 · pp. 1–2, Figure 1 · p. 2
• After SFT enables zero-shot reasoning to generate low-level action tokens, GRPO refines reasoning by sampling trajectories and maximizing an action-accuracy reward.This directly aligns MLLM reasoning with VLA action generation.Source: §1 · p. 2
• ROBOALIGN improves over baseline SFT-only models by 17.5% on LIBERO, 18.9% on CALVIN, and 106.6% in real-world setup, using less than 1% additional RL data.The evaluation attaches a diffusion-based action head to a frozen MLLM backbone and fine-tunes it to generate low-level actions.Source: §1 · p. 2
• On LIBERO, ROBOALIGN’s 17.5% improvement exceeds 13.1% for high-level action prediction and 15.2% for point-trajectory prediction.These are comparisons with other alignment approaches reported for the LIBERO evaluation.Source: §1 · p. 2
• ROBOALIGN achieves state-of-the-art embodied-reasoning results, outperforming GPT-4o and RoboBrain2.0 while preserving general image-understanding performance.The evaluation covers general image understanding, spatial reasoning, and embodied reasoning; the paper contrasts this with ECoT, which often degrades performance on embodied tasks.Source: §1 · p. 2

Multimodal-large-language models for robot control

• MLLM robot-control work spans high-level planning and increasingly precise action generation, motivating direct alignment between MLLM representations and low-level robot actions.Prior approaches generate atomic skills or high-level plans, produce policies end to end, or train action experts on latent states.Source: §2 · p. 2

Multimodal-large-language model for embodied reasoning

• Embodied-reasoning work improves spatial and temporal understanding with VQA-style supervision, but does not directly optimize low-level action generation.Spatial approaches train on 3D scene understanding or predict affordance-related boxes and points; temporal approaches extract high-level actions or object trajectories from videos to construct VQA.Source: §2 · pp. 2–3

Encouraging reasoning through reinforcement learning

• CoT reinforcement learning optimizes reasoning traces against final-answer rewards, while RoboAlign grounds RL in low-level action prediction to align MLLM representations with robot control.The cited paradigm has shown stronger performance and generalization than SFT across domains with substantially less data, sometimes a single example.Source: §2 · p. 3

FAST action tokenization

• FAST tokenization (Pertsch et al., 2025) compresses action chunks into discrete MLLM tokens through DCT, coefficient quantization, flattening, and byte-pair encoding.Each action represents end-effector position, orientation, and gripper state; a horizon-H sequence is transformed into tokens drawn from 2K special tokens added to the MLLM vocabulary.Source: §3 · p. 3

Encouraging reasoning with GRPO

• GRPO (Shao et al. 2024) trains the model to produce <think> reasoning before final answers, using sampled-response rewards for format correctness and answer accuracy with a KL penalty.Rewards are normalized to compute advantages, and the policy is optimized against a reference policy.Source: §3 · p. 3

4. ROBOALIGN: Align Embodied Reasoning with Low-level Actions

• ROBOALIGN addresses MLLMs’ inability to generate specialized FAST actions zero-shot with a two-stage SFT-then-RL pipeline that aligns reasoning and low-level actions.The framework first equips the MLLM to predict FAST tokens through zero-shot reasoning, then uses RL to strengthen reasoning and token accuracy while aiming to preserve general MLLM capabilities.Source: §4 · pp. 3–4, Figure 2 · p. 4

4.1. Stage 1: Integrating Low-level Action with MLLM using SFT

• Stage 1 uses robotics-aware SFT to teach FAST-token generation while preserving general vision-language capabilities and strengthening embodied reasoning.Its data mixture combines open-source embodied-reasoning and general-understanding data, custom robotic VQA, specialized zero-shot reasoning data, and FAST-token robot data. The VQA pipeline uses robot images and metadata to generate QA, captioning, and grounding pairs; reasoning trajectories are filtered by reward and correctness checks, and the paper reports that this data transfers zero-shot reasoning to FAST generation. FAST examples pair BridgeV2 robot images with instructions and target token sequences.Source: §4.1 · pp. 4–5, Table 1 · p. 5

4.2. Stage 2: Aligning Embodied Reasoning with Low-level Action via RL

• Stage 2 applies GRPO to FAST-token examples prompted for explicit reasoning, optimizing both reasoning-format compliance and action-token accuracy.The reward is the arithmetic mean of a binary format reward and an accuracy reward measuring FAST-token prediction; this encourages correctly formatted and accurate sequences.Source: §4.2 · p. 5

5.1. Experimental Setup

• VLAs use the same conversion pipeline: a newly initialized diffusion action head is trained on benchmark data while the MLLM backbone stays frozen.The setup uses Qwen2.5VL-7B-Ins and evaluates on LIBERO, CALVIN, and real-robot tasks.Source: §5.1 · pp. 5–6

5.2. Main Results

• 18.1% → 22.2% CALVIN length-5 success and 70% LIBERO Long success show ROBOALIGN’s gains on long-horizon tasks; LIBERO Goal rises from 42.4% → 87.2%.ROBOALIGN combines SFT and RL; the RL stage uses 12.8K samples, less than 1% of the 2.28M samples used for SFT, and accounts for most of the gain.Source: Table 3 · p. 6, Table 2 · p. 6, §5.2 · p. 7

5.1. Experimental Setup

• For SFT, 1.88M MLLM-task samples plus 400K BridgeV2 FAST-token samples yield 2.28M total; RL uses a 12.8K subset of that FAST-token data.Baselines train only on MLLM data or only on the full BridgeV2 dataset (1.88M samples), for one epoch under ROBOALIGN’s SFT schema. Each benchmark’s provided training data is used to train VLAs from scratch; CALVIN training uses environments A, B, and C, followed by zero-shot evaluation in environment D.Source: §5.1 · pp. 5–6

5.3. Ablation Study and Analyses

• ROBOALIGN improves success in real-robot pick-and-place tasks and generalizes to Qwen3-VL-8B-Ins, with particularly significant gains in LIBERO Long.Real-robot evaluation uses four object tasks and 96 trials per task; the alternate-backbone experiment uses 5K RL samples.Source: Table 4 · p. 7, §5.3 · pp. 7–8, Table 5 · p. 7

5.2. Main Results

• MLLM-only training raises CALVIN length-1 success (77.8% →87.4%) and LIBERO Goal (42.4% →67.8%) but lowers CALVIN length-5 success (18.1% →15.3%).VLA-only training raises LIBERO Goal from 42.4% →82.8% but reduces long-horizon performance; the authors hypothesize this reflects forgetting of general MLLM capabilities.Source: §5.2 · p. 7

5.3. Ablation Study and Analyses

• Low-level action-based RL alignment yields the largest LIBERO gains and uniquely improves long-horizon tasks, unlike language-action or 2D-trajectory RL alternatives.The SFT-based ECoT alignment degrades LIBERO performance relative to ROBOALIGN; the authors attribute this to limited transfer from BridgeV2 and forgetting during SFT.Source: Table 6 · p. 8, §5.3 · p. 8, Table 7 · p. 8
• 69.79% KNN accuracy with ROBOALIGN (SFT+RL), versus 39.06% for both baselines, indicates more discriminative MLLM representations of underlying states.The KNN test uses MLLM representations alone, without robot-state information; the authors connect finer-grained state encoding to accurate action generation.Source: Table 8 · p. 8, §5.3 · p. 8
• ROBOALIGN outperforms specialized embodied-reasoning models on embodied tasks while maintaining strong general MLLM benchmark performance.The authors report that RL alignment with low-level actions does not degrade MLLM capabilities and instead improves them.Source: §5.3 · p. 8

6. Conclusion

• ROBOALIGN uses reinforcement learning and embodied reasoning to improve low-level action prediction by aligning MLLM representations with action policies.The framework is tailored to vision-language-action models.Source: §6 · p. 8
• Across diverse robotic environments and MLLM benchmarks, ROBOALIGN delivers substantial gains in embodied reasoning for MLLM and VLA tasks.The conclusion reports gains across both short- and long-horizon tasks.Source: §6 · p. 8
• Language-only embodied-reasoning fine-tuning yields limited or degraded performance on complex scenarios.Source: §6 · p. 8

• VLAs trained on MLLMs aligned with unsafe action trajectories may be biased toward generating unsafe actions, so this risk should be prevented during training because the MLLM’s training data can be difficult to determine after deployment.Source: Impact Statement · p. 9

A.1. Computing Cost

• MLLM training takes about 30 hours for SFT and 1 hour for reinforcement learning on 8×H200 GPUs; VLA training uses 2×A100 GPUs and takes about 1 hour per 10K steps.Source: §A.1 · p. 13

A.2. Implementation Details for VLA Training

• The implementation uses batch size 32 and trains new action experts for 60K LIBERO, 100K CALVIN, and 30K real-robot steps.Policy fine-tuning hyperparameters follow the official GR00T-N1.5 implementation unless otherwise specified.Source: §A.2 · p. 13

A.3. Training Datasets

• The SFT training set contains 1.88M QA samples spanning general multimodal capability, embodied reasoning, and robot-specific reasoning.It includes 100K LLaVA-OneVision samples, 300K RefSpatial, 200K RoboPoint, 50K EgoPlan-IT, 500K multi-view instructions, and additional robot-focused QA data.Source: §A.3 · p. 13
• Reasoning distillation yields a 76K-sample dataset, while FAST prediction uses 400K BridgeV2 samples and RL uses a 12.8K subset.The reasoning data includes 50K ROBOALIGN VQA and 50K SAM2-derived samples; 30K train the distillation model before augmentation and correctness filtering.Source: §A.3 · p. 13

A.4. Analysis Setup Details

• For embodied-alignment comparisons, all models use RL on the same BridgeV2 images, with separate rewards for high-level actions and 2D trajectory prediction.High-level movements are converted to multiple-choice QA and scored for correctness; the ShareRobot trajectory comparison uses its 6K samples and ThinkAct’s reward formulation.Source: §A.4 · p. 13
• The SFT-based alignment comparison keeps FAST tokens as the action space and trains both methods on the same 12.8K samples atop ROBOALIGN SFT.Both receive one SFT epoch with identical hyperparameters before conversion to VLAs and evaluation in LIBERO.Source: §A.4 · p. 13
• Representation analysis tests whether vision and task instruction alone let the MLLM recover timestep classes from hidden representations.It uses 20 trajectories from one LIBERO long-horizon task, assigns timesteps to 32 DTW classes, and evaluates with a k = 5 KNN classifier.Source: §A.4 · p. 13
• ROBOALIGN (SFT) performs on par with specialized embodied-reasoning MLLMs, and RL further boosts performance across the overall MLLM benchmarks.The evaluation covers MMStar, Robospatial-Home, Where2Place, depth-related BLINK components, and Robot-R1 Bench, following official protocols.Source: §A.4 · pp. 13–14, Table 9 · p. 14

A.5. RL Training Process

• RL steadily raises accuracy reward while response length stays stable, and changes broad plans into concrete actions with situational awareness.Before RL, responses give high-level plans; afterward, they describe immediate actions such as aligning and closing the gripper to pick up cloth.Source: §A.5 · p. 14, Figure 6 · p. 15, Figure 7 · p. 15