PaperlayerView exact revision on arXiv

Source-linked AI summary

OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoning

Fanqi Lin, Ruiqian Nai, Yingdong Hu, Jiacheng You, Junming Zhao, Yang Gao

arXiv:2505.11917v2cs.RO

TL;DR

Robotic systems that separate reasoning from action can have limited mutual awareness and latency. OneTwoVLA unifies both with adaptive reasoning, and experiments report capabilities across planning, recovery, interaction, and visual grounding.

  • Problem

    Dual-system robot approaches separate reasoning from action, with limited mutual awareness between systems and latency remaining challenges for long-horizon manipulation.

    Source: §2 · p. 3

  • Method

    OneTwoVLA unifies reasoning and action in one model that adaptively chooses when to reason or act, and co-trains with synthetic reasoning-centric vision-language data.

    Source: §1 · p. 2, §4.4 · p. 9

  • Results

    OneTwoVLA demonstrates planning, error recovery, human-robot interaction, and visual-grounding capabilities; OneTwoVLA-VL achieves a 73% success rate in open-world visual grounding.

    Source: §1 · p. 2, §4.4 · p. 9

  • Takeaways & Limitations

    OneTwoVLA-VL handles novel interaction scenarios, including switching tasks in response to a human interruption.

    Source: §4.3 · p. 8

    Reasoning triggers rely on hand-selected heuristics, and the robot pauses for two to three seconds while reasoning occurs.

    Source: §5 · p. 9

Abstract

from arXiv · show

General-purpose robots capable of performing diverse tasks require synergistic reasoning and acting capabilities. However, recent dual-system approaches, which separate high-level reasoning from low-level acting, often suffer from challenges such as limited mutual understanding of capabilities between systems and latency issues. This paper introduces OneTwoVLA, a single unified vision-language-action model that can perform both acting (System One) and reasoning (System Two). Crucially, OneTwoVLA adaptively switches between two modes: explicitly reasoning at critical moments during task execution, and generating actions based on the most recent reasoning at other times. To further unlock OneTwoVLA's reasoning and generalization capabilities, we design a scalable pipeline for synthesizing embodied reasoning-centric vision-language data, used for co-training with robot data. We validate OneTwoVLA's effectiveness through extensive experiments, highlighting its superior performance across four key capabilities: long-horizon task planning, error detection and recovery, natural human-robot interaction, and generalizable visual grounding, enabling the model to perform long-horizon, highly dexterous manipulation tasks such as making hotpot or mixing cocktails.

1 INTRODUCTION

The paper aims to give robots the flexible reasoning–acting synergy in which reasoning guides action and action feedback informs further reasoning. It introduces OneTwoVLA, a unified model designed to realize that synergy through adaptive reasoning and action.

Motivation

  • Prior dual-system approaches separate slow high-level reasoning from low-level action, risking capability mismatches and delayed, outdated guidance (Shi et al., 2025).The paper motivates unification as a way to strengthen reasoning–acting synergy.Source: §1 · p. 2

OneTwoVLA

  • OneTwoVLA is a single vision-language-action model that reasons at key moments and otherwise generates actions informed by its most recent reasoning.Key moments include subtask completion, error detection, or a need for human input; reasoning outputs can include scene descriptions, plans, historical summaries, and next-step instructions.Source: §1 · p. 2
  • A scalable pipeline synthesizes high-quality embodied reasoning-centric vision-language data for co-training, which the paper says enhances reasoning and generalization.The unified model naturally supports co-training with vision-language data.Source: §1 · p. 2

Experiments

  • OneTwoVLA outperforms flat VLA by 30% and dual-system VLA by 24% on long-horizon planning, while supporting novel-instruction planning, error recovery, adaptive interaction, and generalizable visual grounding.The reported capabilities include responding to human intervention, seeking clarification under ambiguity, and grounding spatial, attribute, and semantic features for objects absent from robot training data.Source: §1 · p. 2

2 RELATED WORK

Prior VLAs handle varied manipulation tasks but struggle with reasoning and out-of-distribution generalization; this section motivates a unified model that adaptively chooses when to reason or act.

Vision-Language-Action Models

Reasoning for Robot Control

3 METHOD

OneTwoVLA unifies reasoning and action in a policy that chooses between the two modes, trained with robot demonstrations and scalable synthetic embodied-reasoning data.

3.1 FRAMEWORK OF ONETWOVLA

  • In reasoning mode, the policy uses current and reference images, the instruction, and latest reasoning to generate updated text; in acting mode, it also uses proprioception to produce action chunks.The reasoning-mode inputs include images from multiple cameras and reference images from the latest reasoning timestep.Source: §3.1 · p. 3
  • With [BOR] and [BOA] decision tokens, OneTwoVLA acts primarily and reasons at critical steps, matching flat VLA π0 (Black et al., 2024) task time while avoiding the latency of always-reasoning Hi Robot (Shi et al., 2025) and ViLa (Hu et al., 2023).When it detects an error, it reasons about a corrective strategy; interaction text is added to the instruction for subsequent steps.Source: §3.1 · pp. 3–4

3.2 CURATING ROBOT DATA WITH EMBODIED REASONING

  • Demonstrations are divided into reasoning intervals, annotated with reasoning, and acting intervals; decision-token supervision uses [BOR] when reasoning is stale and [BOA] once updated or during acting.Reasoning covers subtask completion, error detection, and human interaction; its four fields are scene description, high-level plan, historical summary, and immediate next step, with failure-recovery data and interaction context also annotated.Source: §3.2 · p. 4
  • 81.5% of Tomato-Egg intervals were judged correct and 83.3% of scene descriptions reasonable; the automated pipeline samples 32 frames, labels post-subtask intervals with Gemini 2.5, and generates four reasoning fields from midpoint frames.The fields comprise a scene description, high-level plan, historical summary, and next step.Source: §3.2 · pp. 4–5

3.3 SCALABLE SYNTHESIS OF VISION-LANGUAGE DATA WITH EMBODIED REASONING

  • A three-step pipeline uses Gemini 2.5 Pro to describe tabletop layouts, FLUX.1-dev to synthesize images with visual augmentations, and Gemini to generate instructions and reasoning, producing 16,000 samples.Augmentations randomly apply fisheye distortion or composite a robot gripper with adaptive brightness.Source: §3.3 · p. 5
  • Synthetic tasks cover visual grounding, where reasoning names the implicitly referenced object, and long-horizon objectives, where reasoning provides a step-by-step plan; some plans include human-robot interaction.Visual references may use spatial relationships, attributes, or semantic features; reasoning may optionally give the object’s location.Source: §3.3 · p. 5

4 EXPERIMENTS

Real-world experiments evaluate OneTwoVLA on long-horizon planning, error recovery, human-robot interaction, and visual grounding. They also test whether co-training with synthetic vision-language data improves generalization to novel tasks, scenarios, and objects.

4.1 LONG-HORIZON TASK PLANNING

  • Across three challenging long-horizon tasks, OneTwoVLA achieves an average success rate of 87%, outperforming π0 by 30% and the dual-system approach by 24%; it consistently generates correct plans, tracks task progress, and outputs precise actions.The dual-system approach sometimes issued infeasible atomic commands, and Gemini 2.5 Pro’s inference latency could delay reasoning updates.Source: §4.1 · pp. 6–7, Figure 6 · p. 7
  • OneTwoVLA-VL generalized to novel planning tasks, while π0 and OneTwoVLA without VL co-training showed minimal generalizable planning ability.OneTwoVLA-VL was co-trained on robot data and 16,000 synthetic VL samples, then evaluated on instructions absent from the robot data; examples included retrieving hidden cola, using a tool, and inferring a drink from human intent.Source: Figure 6 · p. 7, §4.1 · p. 7

4.2 ERROR DETECTION AND RECOVERY

  • For Hotpot and Tomato-Egg recovery errors, OneTwoVLA reasoned through corrective actions, whereas π0 often ignored errors and the dual-system approach responded too late.Recovery demonstrations taught the model to realign and retry a strainer grasp or adjust its grip on a slipping oil bottle; delayed detection could leave the dual-system robot in an out-of-distribution state.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 NATURAL HUMAN-ROBOT INTERACTION

  • In 20 novel interaction scenarios, OneTwoVLA-VL achieved 72.5% success and handled clarification and interruptions, while OneTwoVLA without VL co-training struggled to interpret unseen commands.In standard Hotpot and Cocktail interactions, OneTwoVLA also handled human interventions coherently, while the dual-system baseline could lose task context; π0 was omitted because it cannot generate language outputs.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 ENHANCED VISUAL GROUNDING

  • OneTwoVLA reached a 78% Single-Env success rate versus π0’s 5%, and OneTwoVLA-VL achieved 73% in Open-World visual grounding.Open-World evaluation included unseen environments and objects, including objects absent from both robot and VL training data; OneTwoVLA-VL generalized to examples such as Sprite and GoPro. The study tests grounding of spatial relationships, attributes, and semantic features.Source: §4.4 · pp. 8–9, Table 3 · p. 9

5 CONCLUSION, LIMITATIONS AND FUTURE WORK … C TASKS AND EVALUATIONS

OneTwoVLA unifies reasoning and acting in one model that switches adaptively between modes, supported by reasoning-enriched robot data and a scalable pipeline for embodied reasoning-centric vision-language data. Experiments demonstrate strengths in long-horizon planning, error detection and recovery, natural human-robot interaction, and visual grounding.

5 CONCLUSION, LIMITATIONS AND FUTURE WORK

  • OneTwoVLA unifies reasoning and acting with adaptive mode switching, and experiments show strengths in planning, error recovery, human-robot interaction, and visual grounding.The framework is supported by reasoning-enriched robot data and a scalable pipeline for synthesizing embodied reasoning-centric vision-language data.Source: §5 · p. 9
  • Handselected heuristics choose reasoning steps; future work could use reinforcement learning, while asynchronous architectures could reduce the robot’s two-to-three-second reasoning pauses.The authors compare heuristic selection to supervised fine-tuning that aligns policy with candidate reasoning steps and content.Source: §5 · p. 9
  • Action-inference costs may bottleneck larger models, and resource constraints limited evaluation to high-quality synthetic vision-language data; future work could improve inference and test other data sources.The authors anticipate that advanced inference techniques from LLM research may accelerate action inference.Source: §5 · p. 9

APPENDIX

  • The appendix directs readers to the website for robot rollout videos.The supplied passage gives the URL https://one-two-vla.github.io/.Source: APPENDIX · p. 16

A LARGE LANGUAGE MODELS USAGE STATEMENT

C.1 LONG-HORIZON TASKS

The section describes three long-horizon robot tasks—Tomato-Egg, Hotpot, and Cocktail—and the demonstrations and recovery or interaction behaviors used to evaluate them.

Tomato-Egg

  • Tomato-Egg preparation involves cooking and plating a scramble, with dedicated data enabling recovery when the oil bottle slips from the gripper; 200 demonstrations were collected.The robot pours oil, tomato, and egg liquid into a cooking machine, then scoops out the finished scramble and plates it.Source: §C.1 · p. 17

Hotpot

  • In Hotpot, the robot sequentially dips beef and a requested vegetable, with recovery data for failed strainer grasps; 600 demonstrations were collected.In 10 experiments for OneTwoVLA and the dual-system approach, the robot asks which vegetable to dip after the human adds a request while the beef cooks; π0 omits this interaction because it lacks text output.Source: §C.1 · p. 17

Cocktail

  • Cocktail preparation covers three drinks, with 300 demonstrations total, and tests clarification of general requests and response to an ingredient substitution interruption.In 10 trials, OneTwoVLA and the dual-system approach ask which cocktail the human wants; in three Vodka Sunrise trials, the robot switches from orange to lemon vodka on request. π0 omits clarification.Source: §C.1 · p. 17

C.2 GENERALIZABLE PLANNING TASKS

Co-training with synthesized embodied reasoning-centric vision-language data enables OneTwoVLA to generalize beyond short-horizon atomic-skill demonstrations to unseen, long-horizon planning tasks. The tasks test inference from user intent and scene context, sequential object handling, and tool use.

C.2 GENERALIZABLE PLANNING TASKS

  • Co-training synthesized embodied reasoning-centric vision-language data with robot demonstrations equips OneTwoVLA to generalize to previously unseen tasks beyond short-horizon atomic skills.The demonstrations cover atomic skills such as pick, place, move, open, close, and pour, and were collected using single-arm Franka and dual-arm ARX systems.Source: §C.2 · p. 18
  • For “Get me a can of icy cola,” the robot infers the cola is in the fridge and plans to open it, locate the can, and retrieve it.The cola can is not directly visible in the scene.Source: §C.2 · p. 18
  • For “Pass me an empty plate,” the robot removes the apples and grapes from the nonempty plate before picking it up.Source: §C.2 · p. 18
  • For an out-of-reach cocoa powder can, the robot grasps a nearby stick, sweeps the can within reach, and then picks it up.Source: §C.2 · p. 18
  • For drink preparation, the robot maps user intent to coconut latte for staying awake, kale juice for health, or a blue mood cocktail when feeling down.This task requires scene-aware understanding of user intent.Source: §C.2 · p. 18

C.3 VISUAL GROUNDING TASKS · D MORE REASONING EXAMPLES

The visual-grounding section specifies instruction data and evaluation coverage, while the reasoning-examples section describes annotation and assembly for recovery and dialogue. Together, they show how reasoning examples span task types and interaction conditions.

C.3 VISUAL GROUNDING TASKS

  • Single-Env demonstrations receive 11 instruction–reasoning pairs and Open-World demonstrations 17, covering direct names, spatial relations, attributes, and semantic features.Gemini 2.5 Pro generates the pairs, which human annotators verify.Source: §C.3 · p. 18
  • Each method is tested 40 times in each setting—10 tests per reference type—with results broken down across the four instruction-reference types.The reported types are direct names, spatial relationships, object attributes, and semantic features.Source: §C.3 · p. 18, Table 4 · p. 19
  • Open-World evaluation covers 20 objects: five seen in robot data, ten present only in synthetic vision-language data, and five unseen in either dataset; Single-Env uses four objects.The Single-Env objects are a blue cube, eggplant toy, coconut water bottle, and black mouse.Source: §C.3 · p. 19
  • Open-World visual grounding uses 16 training environments and eight evaluation environments.The environments are depicted in Figures 11 and 12, respectively.Source: §C.3 · p. 19, Figure 11 · p. 20, Figure 12 · p. 21

D MORE REASONING EXAMPLES

  • The reasoning examples cover long-horizon and generalizable planning, error detection and recovery, natural human–robot interaction, and both visual-grounding settings.The section presents examples for each of these six task categories.Source: D MORE REASONING EXAMPLES · p. 22
  • Prompt templates support both reasoning-interval annotation and reasoning-content generation.The section provides separate templates for the two stages.Source: D MORE REASONING EXAMPLES · p. 22, Figure 13 · p. 28, Figure 14 · p. 29
  • For error recovery, plan augmentation prompts reasoning intervals around failures and recovery steps, while assembly retains the original plan and conditions retry actions on the failed step.A failed step may be followed by a retry, optionally preceded by retraction.Source: D MORE REASONING EXAMPLES · p. 22
  • For human–robot interaction, explicit dialogue substeps guide annotation, and the reasoning tuple records the robot’s question and human’s answer to condition the updated plan and next action.The historical summary records newly acquired information; the answer may be empty when no interaction occurs.Source: D MORE REASONING EXAMPLES · p. 22

E SYNTHETIC VISION-LANGUAGE DATA EXAMPLES · F IMPLEMENTATION DETAILS

The appendix documents a fully automated synthetic-data pipeline for visual grounding and long-horizon planning, with examples, image augmentations, generation prompts, and sampled quality checks.

E SYNTHETIC VISION-LANGUAGE DATA EXAMPLES

F.1 ROBOT DATA INTERVALS

OneTwoVLA trains on separate reasoning and acting intervals, with different targets for updating reasoning, producing actions, and choosing when to reason. Deployment uses reasoning at only a small number of steps to preserve operational efficiency.

F.1 ROBOT DATA INTERVALS

  • In reasoning intervals, OneTwoVLA predicts an updated reasoning state and actions conditioned on either that update or the latest reasoning.Training includes [BOR] with updated reasoning and [BOA] with updated reasoning, plus actions from the latest reasoning without [BOA] supervision to guard against delayed reasoning updates.Source: §F.1 · p. 37
  • In acting intervals, OneTwoVLA predicts [BOA] and actions from the latest reasoning, with optional [BOR] training to counter underprediction of reasoning decisions.The optional objective predicts [BOR] from outdated reasoning without supervising reasoning content, addressing the imbalance caused by longer acting intervals and deployment failures to enter reasoning mode.Source: §F.1 · p. 37
  • At deployment, the robot reasons at only a small number of steps rather than in continuous intervals, keeping overall operational efficiency almost unaffected.Source: §F.1 · p. 37

F.2 POLICY TRAINING · F.3 DEPLOYMENT · G OTHER FINDINGS

OneTwoVLA’s deployment keeps acting efficient while reserving reasoning for critical moments, and its training and deployment inputs and action-smoothing mechanisms support coherent execution. It can also produce multiple valid actions when an instruction leaves object choice unspecified.

F.2 POLICY TRAINING

  • For each task, OneTwoVLA trains from π0 for 30,000 steps on 8xH100 GPUs, taking approximately 10 hours and using π0’s training hyperparameters.The model uses the current image and reference image as observations, because textual scene descriptions may become outdated as the task progresses.Source: §F.2 · p. 37
  • OneTwoVLA also receives robot proprioceptive states from 0.05 and 0.25 seconds earlier, providing temporal context for smoother, more consistent actions.The reference image corresponds to the image observation for the current reasoning content and helps prevent confusion from outdated textual descriptions.Source: §F.2 · p. 37

F.3 DEPLOYMENT

  • At deployment, temporal ensembling averages overlapping action predictions with exponential weighting; in acting mode, new sequences are generated every 0.2 seconds.The method is used to ensure smooth action execution.Source: §F.3 · p. 37
  • Acting-mode computation remains well below 0.2 seconds; reasoning under 20 tokens leaves efficiency unaffected, while reasoning over 100 tokens can pause the robot for seconds.Table 14 reports computation-time measurements for π0 and OneTwoVLA on an NVIDIA 4090 GPU with two image inputs.Source: §F.3 · p. 37, Table 14 · p. 38
  • Reasoning took 16 of 183 seconds (8.7%) in one Tomato-Egg trial and 14 of 135 seconds (10.4%) in one Preparing Mountain Fuji trial.In each trial, reasoning occurred five times.Source: §F.3 · p. 38

G OTHER FINDINGS

  • When an instruction does not specify which of two symmetrically placed cubes or bottles to grasp, OneTwoVLA can reason to choose either object and produce multimodal actions.Figure 19 illustrates this behavior.Source: Figure 19 · p. 39

G.1 ONETWOVLA PRODUCES MULTI-MODAL ACTIONS … I FAILURE CASES

Across these sections, OneTwoVLA produces varied actions for ambiguous grasps and keeps actions aligned with its reasoning; the paper also describes two robot platforms and reports manipulation and generalization failures.

G.1 ONETWOVLA PRODUCES MULTI-MODAL ACTIONS

  • OneTwoVLA’s probabilistic reasoning supports varied left-right grasp decisions, while π0 consistently selects right-side objects and exhibits unimodal behavior.The paper attributes π0’s behavior to its direct use of the ambiguous instruction, whereas OneTwoVLA demonstrations include disambiguating reasoning.Source: §G.1 · p. 39
  • π0’s validation action mean squared error (MSE) is 56% higher than OneTwoVLA’s.The paper also reports that π0 occasionally fails to grasp the block, while OneTwoVLA achieves precise grasps consistently.Source: §G.1 · p. 39

G.2 ONETWOVLA PRODUCES REASONING-COMPLIANT ACTIONS

  • OneTwoVLA’s actions follow its reasoning even when that reasoning is wrong, such as reaching toward incorrectly located food or an object mismatched to the instruction.The paper describes this as unified cognition and behavior and suggests that better reasoning—through more vision-language data, a stronger VLM, or more precise annotations—may support more appropriate actions (Zawalski et al., 2024).Source: §G.2 · pp. 39–40

H HARDWARE SETUP

  • The platforms are a single 7-DoF Franka arm with a 7-dimensional action space and two 6-DoF ARX arms with a 14-dimensional action space.The Franka has a wrist-mounted GoPro and uses UMI demonstrations; the ARX setup has three cameras and uses Meta Quest teleoperation.Source: H HARDWARE SETUP · p. 40

I FAILURE CASES

  • Failures include dropping the tomato-egg plate, targeting enoki mushrooms instead of bok choy, spilling orange juice, and approaching a chessboard instead of the instructed basketball toy.The paper suggests that larger robot datasets and richer vision-language data could improve fine-grained actions and generalization.Source: I FAILURE CASES · p. 41, Figure 21 · p. 41

Related papers

Loading 2505.11917v2…