PaperlayerView exact revision on arXiv

Source-linked AI summary

OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoning

Fanqi Lin, Ruiqian Nai, Yingdong Hu, Jiacheng You, Junming Zhao, Yang Gao

arXiv:2505.11917v2cs.RO

TL;DR

Dual-system robot controllers face coordination and latency limitations when reasoning and acting are separated. OneTwoVLA unifies both capabilities in one model, adaptively reasoning at critical moments and co-training with synthetic embodied reasoning-centric vision-language data. Experiments show strong performance across long-horizon planning, error recovery, human-robot interaction, and visual grounding.

  • Problem

    Separate reasoning and acting systems can lack mutual awareness of capabilities and suffer latency, limiting synergistic robot control.

  • Method

    OneTwoVLA is a unified vision-language-action model that adaptively switches between explicit reasoning and action generation, co-trained with synthetic embodied reasoning-centric vision-language data.

  • Results

    OneTwoVLA demonstrates strong performance across long-horizon planning, error detection and recovery, human-robot interaction, and generalizable visual grounding.

  • Takeaways & Limitations

    The unified adaptive model supports long-horizon, dexterous manipulation while integrating reasoning, acting, interaction, and visual grounding.

  • Takeaways & Limitations

    Reasoning-step selection relies on handselected heuristics, and the robot pauses two to three seconds during reasoning.

Abstract

from arXiv · show

General-purpose robots capable of performing diverse tasks require synergistic reasoning and acting capabilities. However, recent dual-system approaches, which separate high-level reasoning from low-level acting, often suffer from challenges such as limited mutual understanding of capabilities between systems and latency issues. This paper introduces OneTwoVLA, a single unified vision-language-action model that can perform both acting (System One) and reasoning (System Two). Crucially, OneTwoVLA adaptively switches between two modes: explicitly reasoning at critical moments during task execution, and generating actions based on the most recent reasoning at other times. To further unlock OneTwoVLA's reasoning and generalization capabilities, we design a scalable pipeline for synthesizing embodied reasoning-centric vision-language data, used for co-training with robot data. We validate OneTwoVLA's effectiveness through extensive experiments, highlighting its superior performance across four key capabilities: long-horizon task planning, error detection and recovery, natural human-robot interaction, and generalizable visual grounding, enabling the model to perform long-horizon, highly dexterous manipulation tasks such as making hotpot or mixing cocktails.

1 INTRODUCTION

OneTwoVLA unifies reasoning and acting in a single model, adaptively invoking language reasoning at critical moments and using recent reasoning to guide actions otherwise. The paper combines this architecture with scalable embodied reasoning-centric vision-language data and evaluates four capabilities.

  • Human physical intelligence interleaves reasoning and acting, with reasoning guiding actions and actions providing feedback for subsequent reasoning.
  • Dual-system approaches separate slow high-level reasoning from low-level action generation, creating challenges in coordination and latency.
  • OneTwoVLA is a unified vision-language-action model that performs both System One acting and System Two reasoning.
  • The model triggers textual reasoning at critical steps, then generates actions informed by its most recent reasoning at other times.
  • Experiments report gains in long-horizon planning, error recovery, human-robot interaction, and generalizable visual grounding.

2 RELATED WORK

Related work uses pretrained vision-language models for high-level reasoning and vision-language-action policies for low-level control. OneTwoVLA addresses coordination and efficiency limitations by adaptively deciding when its unified model should reason or act.

  • Vision-language-action models use pretrained vision-language models and large robot datasets to support general-purpose robotic control.
  • Reasoning-oriented robot-control studies show that high-level reasoning can enhance low-level policy performance.
  • Dual-system frameworks assign high-level reasoning to a foundation model and action generation to a separate low-level policy.
  • Separate systems can lack mutual awareness, while extensive or delayed reasoning introduces latency and may leave the acting policy in out-of-distribution states.
  • OneTwoVLA adaptively decides when to reason versus act, targeting both informative reasoning and inference efficiency.

3 METHOD

OneTwoVLA uses a unified policy that chooses between reasoning and action modes, trains on curated reasoning-annotated robot data, and scales supervision with synthetic embodied reasoning-centric vision-language data.

  • 3.1 FRAMEWORK OF ONETWOVLA: The policy predicts whether to enter reasoning mode or acting mode at each timestep from camera observations, the instruction, and the latest reasoning content.
  • 3.1 FRAMEWORK OF ONETWOVLA: In reasoning mode, the model generates updated textual reasoning; in acting mode, it generates an action chunk using the latest reasoning and robot proprioceptive state.
  • 3.1 FRAMEWORK OF ONETWOVLA: Special decision tokens [BOR] and [BOA] select reasoning and action generation, respectively, while reference-image histories help prevent ambiguous states.
  • 3.1 FRAMEWORK OF ONETWOVLA: The framework acts primarily and invokes reasoning at a few critical steps, matching flat-VLA completion times while avoiding the extensive reasoning latency of always-reasoning systems.
  • 3.2 CURATING ROBOT DATA WITH EMBODIED REASONING: Curated robot data segments demonstrations into reasoning intervals and action intervals to teach synergistic reasoning and acting.
  • 3.2 CURATING ROBOT DATA WITH EMBODIED REASONING: Embodied reasoning content contains a scene description, high-level plan, historical summary, and immediate next step.
  • 3.2 CURATING ROBOT DATA WITH EMBODIED REASONING: The automated annotation pipeline achieves 81.5% human-judged interval correctness and 83.3% reasonable scene descriptions on Tomato-Egg.
  • 3.3 SCALABLE SYNTHESIS OF VISION-LANGUAGE DATA WITH EMBODIED REASONING: Synthetic vision-language data covers visual grounding and long-horizon tasks, with reasoning that identifies target objects or supplies step-by-step plans.

4 EXPERIMENTS

Real-world experiments evaluate OneTwoVLA across long-horizon planning, recovery, human interaction, and visual grounding. Results show strong task performance and improved generalization from vision-language co-training, while exposing latency and inference-efficiency limitations.

  • Long-Horizon Task Planning: OneTwoVLA achieves an average 87% success rate across three challenging long-horizon tasks, outperforming π0 by 30% and the dual-system approach by 24%.It tracks plans and progress while generating precise actions.
  • Error Detection and Recovery: OneTwoVLA detects execution errors, reasons about recovery, and generates corrective actions for grasping and pouring failures.Recovery behaviors include repositioning the gripper and tightening the grasp before retrying.
  • Natural Human-Robot Interaction: OneTwoVLA-VL achieves a 72.5% success rate on 20 novel human-interaction scenarios, handling clarification requests and dynamic interruptions.Without vision-language co-training, OneTwoVLA fails to interpret such unseen interaction commands.
  • Enhanced Visual Grounding: Explicit reasoning yields 78% success in single-environment visual grounding, compared with 5% for π0, while also reducing action validation MSE by 62%.OneTwoVLA reasons about spatial relationships, attributes, and semantic features before selecting target objects.
  • Enhanced Visual Grounding: OneTwoVLA-VL reaches 73% success in open-world grounding and generalizes to objects unseen in both robot and vision-language training data.The authors attribute this capability to vision-language co-training activating pretrained web knowledge.
  • Limitations: The framework still relies on hand-selected reasoning heuristics, incurs two to three seconds of reasoning pauses, and may face action-inference bottlenecks at larger scales.The paper proposes reinforcement learning, asynchronous architectures, and advanced inference techniques as future directions.

C.1 LONG-HORIZON TASKS

The appendix describes three long-horizon manipulation tasks—Tomato-Egg, Hotpot, and Cocktail—that require sequential actions, dexterous handling, and, for some trials, recovery or dialogue.

  • Task overview: The appendix presents execution processes for Tomato-Egg, Hotpot, and Cocktail as three long-horizon tasks.
  • Tomato-Egg: Tomato-Egg requires sequential pouring, cooking, scooping, transferring, and final utensil placement.Dedicated recovery data addresses oil-bottle slips during pouring.
  • Hotpot: Hotpot requires selecting beef and a vegetable, placing them sequentially into a strainer, waiting for cooking, and lifting it.
  • Cocktail: Cocktail preparation requires pouring three to four ingredients for one of Mojito, Mountain Fuji, or Vodka Sunrise.In some trials, the robot first asks which cocktail the human wants, then follows the specified choice.

C.2 GENERALIZABLE PLANNING TASKS

The generalizable planning evaluation tests whether robots can infer unstated goals and compose multi-step actions for novel instructions, including retrieval, clearing, tool use, and intent-based drink preparation.

  • Evaluation setup: OneTwoVLA-VL co-trains robot demonstrations with 16,000 synthesized vision-language samples and is evaluated on instructions absent from the robot data.
  • Get Icy Cola: Get Icy Cola requires inferring that “icy cola” is stored in the fridge, then opening the fridge, locating the can, and retrieving it.
  • Empty Plate: Empty Plate requires removing apples and grapes before picking up the plate.
  • Tool Use: Tool Use requires grasping a nearby stick, sweeping an out-of-reach cocoa powder can within reach, and then picking it up.
  • Prepare Drinks: Prepare Drinks maps user intent such as staying awake, wanting something healthy, or feeling down to appropriate drinks.

C.3 VISUAL GROUNDING TASKS

The visual grounding evaluation varies how instructions refer to target objects and compares performance across familiar and open-world environments, supported by annotated reasoning data and task-specific prompts.

  • Reference types: Instructions refer to target objects through direct names, spatial relationships, attributes, or semantic features.Single-Env demonstrations use 11 instruction-reasoning pairs, while Open-World demonstrations use 17.
  • Evaluation protocol: Each method is evaluated 40 times, with 10 tests for each of the four reference types.Table 4 reports results separately for direct names, spatial relationships, attributes, and semantic features.
  • Open-World setting: Open-World evaluation uses objects seen in robot data, objects present only in synthetic vision-language data, and objects unseen in both datasets.
  • Environments: The Open-World task uses 16 training environments and 8 evaluation environments.
  • Examples: The appendix provides reasoning examples for Single-Env and Open-World visual grounding alongside examples for planning, recovery, and interaction.
  • Reasoning annotation: Reasoning data are generated through prompt templates for interval annotation and reasoning-content generation, with plan augmentation supporting recovery and interaction scenarios.Recovery inserts failure and retry micro-steps, while interaction inserts question-and-answer turns and conditions subsequent planning on the answer.
  • Interval annotation: Boundary intervals are contiguous frame ranges separating completed previous steps from not-yet-started next steps while keeping the tabletop substantially visible.
  • Prompting: The appendix includes a prompt template for generating scene descriptions and a table of reasoning examples for Open-World visual grounding.

E SYNTHETIC VISION-LANGUAGE DATA EXAMPLES

The synthetic data pipeline generates tabletop images, descriptions, instructions, and embodied reasoning for visual grounding and long-horizon planning, then checks sampled outputs for image and text errors.

  • Dataset construction: The pipeline creates 16,000 synthetic images, including 6,000 visual-grounding instances and 10,000 long-horizon planning instances.The 6,000 grounding images each receive 17 instruction-reasoning pairs.
  • Dataset construction: Synthetic examples pair textual tabletop descriptions and images with instruction-reasoning data for visual grounding and long-horizon tasks.
  • Generation prompts: Gemini 2.5 Pro generates the synthetic images, descriptions, instructions, and reasoning content through dedicated prompts.Prompts cover tabletop descriptions, visual grounding, and long-horizon task instructions and reasoning.
  • Data quality: A random sample of 50 synthetic instances is checked for hallucinated objects, implausible physics, contradictory reasoning, and fully correct cases.
  • Image augmentation: Synthetic images are augmented with fisheye distortion, robot-gripper compositing, adaptive brightness, or both transformations.
  • Task design: Visual-grounding instructions are categorized by spatial, semantic, and attribute properties.
  • Task design: Long-horizon prompts ask for multi-step tasks based on objects in the generated image.

F.1 ROBOT DATA INTERVALS

OneTwoVLA trains on alternating reasoning and acting intervals, learning when to update reasoning and when to generate actions from the latest reasoning. Deployment uses reasoning only at a small number of critical steps to preserve efficiency.

  • Reasoning intervals: During reasoning intervals, the model predicts a reasoning-boundary token, updated reasoning, and actions conditioned on that update.It also learns to act from the latest reasoning without requiring a boundary token, reducing risk when reasoning updates are delayed.
  • Acting intervals: During acting intervals, OneTwoVLA predicts an action-boundary token and actions from the latest reasoning content.It may also predict a reasoning-boundary token from outdated reasoning without supervising the reasoning content.
  • Deployment implication: Training uses reasoning intervals to encourage reasoning-process learning, while deployment invokes reasoning at only a few steps rather than continuously.The paper states that this design keeps overall operational efficiency almost unaffected.

F.2 POLICY TRAINING

For each task, OneTwoVLA is trained from π0 with modified image inputs that include both current and reference observations. The reference image helps preserve the initial scene when textual descriptions become outdated.

  • Training setup: OneTwoVLA uses π0 as its base model and trains each task for 30,000 steps on 8xH100 GPUs, requiring approximately 10 hours.Training hyperparameters are adopted from π0.
  • Input modification: The model receives current and reference images as visual observations.The reference image addresses stale textual scene descriptions as the task progresses.

F.3 DEPLOYMENT

Deployment combines temporally ensembled action generation with sparse reasoning. Acting computation remains below 0.2 seconds, while reasoning occupies a small fraction of long-horizon execution time.

  • Action execution: In acting mode, the policy generates overlapping action sequences every 0.2 seconds and averages predictions with exponential weighting.This temporal ensemble is used to smooth executed actions.
  • Computation: OneTwoVLA’s acting-mode computation remains well below 0.2 seconds despite additional reasoning input and an extra [BOA] output.The computation comparison uses two image inputs on an NVIDIA 4090 GPU.
  • Reasoning overhead: In a Tomato-Egg trial, reasoning took 16 seconds across 5 occurrences, representing 8.7% of the 183-second task.This example quantifies the runtime share of sparse reasoning during a long-horizon task.
  • Reasoning overhead: In a preparing Mountain Fuji trial, reasoning took 14 seconds across 5 occurrences, representing 10.4% of the 135-second task.Reasoning therefore occupied a limited portion of the reported execution duration.
  • Mode outputs: The deployment comparison distinguishes OneTwoVLA-Act outputs of [BOA] from OneTwoVLA-Reason outputs containing [BOR] and updated reasoning.Reasoning-mode timings are shown for output lengths of 20, 100, and 200 tokens.

G.1 ONETWOVLA PRODUCES MULTI-MODAL ACTIONS

OneTwoVLA produces diverse actions under ambiguous instructions by using probabilistic reasoning to select different valid object choices. Its actions remain aligned with its reasoning, while failures expose limits in grounding and manipulation accuracy.

  • Comparative methods: The method resolves ambiguity by adding disambiguating reasoning that specifies whether to pick the left or right object.π0 instead receives the original instruction without explicit disambiguation.
  • Experimental results: OneTwoVLA grasped the left and right cubes 9 and 11 times, and the left and right bottles 8 and 12 times, across 20 trials each.The balanced outcomes demonstrate multimodal action selection in the reported experiments.
  • Experimental results: Its probabilistic reasoning can sample different decisions, whereas π0 consistently selected only right-side objects in these experiments.The reasoning content also helped OneTwoVLA fit actions more accurately than π0 in the reported grasping comparison.
  • Reasoning-compliant actions: OneTwoVLA’s actions consistently align with its reasoning, even when the reasoning is incorrect.Examples include reaching toward incorrectly reasoned food locations in Hotpot and moving to a mismatched object in Open-World tasks.
  • Failure cases: Reported failures include dropped plates, incorrect ingredient selection, inaccurate pouring, and vulnerability to objects absent from training data.The Open-World example shows repeated movement toward a chessboard instead of the instructed small basketball toy.
Loading 2505.11917v2…