PaperlayerView exact revision on arXiv

Source-linked AI summary

LabVLA: Grounding Vision-Language-Action Models in Scientific Laboratories

Baochang Ren, Xinjie Liu, Xi Chen, Yanshuo Liu, Chenxi Li, Daqi Gao, Zeqin Su, Jintao Xing, Zirui Xue, Rui Li, Xiangyu Zhao, Shuofei Qiao, Minting Pan, Wangmeng Zuo, Lei Bai, Dongzhan Zhou, Ningyu Zhang, Huajun Chen

arXiv:2606.13578v2cs.CLcs.AIcs.LGcs.MMcs.RO

TL;DR

Scientific laboratory VLA systems lack laboratory-specific supervision and must handle diverse embodiments, instruments, liquids, and fixed workflows. LabVLA addresses this with RoboGenesis simulation data and a two-stage VLA training recipe, achieving the highest average success rate among evaluated baselines in both in-distribution and out-of-distribution LabUtopia settings.

  • Problem

    Existing VLA policies rarely cover scientific instruments, transparent liquids, fixed laboratory workflows, or the diverse robot embodiments used for protocol execution.

  • Method

    RoboGenesis synthesizes validated, randomized laboratory workflow demonstrations across robot profiles, while LabVLA combines FAST action-token pretraining with flow-matching posttraining under knowledge insulation.

  • Results

    LabVLA achieves the highest average success rate among evaluated baselines, reaching 71.1% ID and 70.0% OOD and outperforming π0 by 7.8 and 6.8 percentage points, respectively.

  • Takeaways & Limitations

    The results distinguish approachable simulated laboratory operations from harder tasks such as liquid pouring that still need better data, control, or recovery.

  • Takeaways & Limitations

    Most validation occurs in simulation, while the real-robot study covers only four benchtop tasks on a single Franka platform.

Abstract

from arXiv · show

Scientific laboratories increasingly rely on AI systems to reason about experiments, but the physical act of doing science remains largely outside their reach. AI can help read literature, generate hypotheses, and plan protocols, yet the execution of those protocols at the bench still requires a human operator. Vision-Language-Action (VLA) models provide one possible interface between written protocols and robot execution, but existing policies are trained mostly on household and tabletop demonstrations and rarely encounter the instruments, transparent liquids, or fixed protocol workflows found in scientific laboratories. Closing this gap requires both laboratory-specific supervision and a unified learning framework that can accommodate the diverse robot embodiments used to execute experimental protocols. We therefore identify data and embodiment as central bottlenecks alongside model design. To address the data side, we build RoboGenesis, a simulation-based workflow and data engine that composes configured laboratory workflows from atomic skills, validates and filters rollouts, and exports structured demonstrations across supported robot profiles. On the policy side, we present LabVLA, trained with a two-stage recipe: FAST action token pretraining first makes the Qwen3-VL-4B-Instruct backbone action aware before any continuous control is learned, and flow matching posttraining then attaches a DiT action expert under knowledge insulation. On the LabUtopia benchmark, LabVLA achieves the highest average success rate among all evaluated baselines under both in-distribution and out-of-distribution settings.

1. Introduction

Scientific laboratory automation poses VLA-specific data and embodiment challenges because protocols require precise, state-sensitive manipulation across diverse robot platforms. The paper addresses these bottlenecks with RoboGenesis, a simulation-based workflow engine, and LabVLA, a protocol-conditioned policy trained with FAST action-token pretraining and flow-matching posttraining.

  • Motivation: Laboratory manipulation requires fine spatial precision, reliable contact control, physical state changes, and execution across heterogeneous robot embodiments.Examples include pipetting, cap screwing, liquid transfer, heating, mixing, color transitions, and container placement.
  • Motivation: Existing robot corpora provide broad manipulation priors, but primarily cover household and tabletop settings rather than scientific laboratory workflows.Open X-Embodiment, DROID, and BridgeData V2 support progress in general robot operation, while OpenVLA and 𝜋0 scale on these corpora.
  • RoboGenesis: RoboGenesis is an Isaac Sim-based synthesis engine that composes laboratory workflows, randomizes scenes, replays protocols across robot profiles, and exports success-filtered demonstrations.Simulation scales data collection with compute rather than instrument, supervision, and safety overhead.
  • LabVLA: LabVLA connects written laboratory protocols to embodied execution through protocol-conditioned synthesis, FAST action-token pretraining, flow-matching posttraining, and a shared cross-embodiment schema.The policy is built on a Qwen3-VL-based backbone and combines the synthesized corpus with broad real-robot pretraining data.
  • Contributions: The work formulates scientific laboratory automation as a VLA learning problem and identifies data and embodiment as central bottlenecks alongside model design.RoboGenesis is designed to produce laboratory demonstrations that existing robot corpora rarely cover.

2. RoboGenesis: A Programmable Workflow and Data Engine

RoboGenesis addresses laboratory VLA data bottlenecks by composing validated, executable scenes and reusable protocol workflows across robot embodiments. It generates diversified, structured demonstrations while preserving the link between written protocols and executed behavior.

  • Motivation: RoboGenesis combines executable laboratory scenes, reusable protocol structure, and task-paired trajectories, addressing limitations of existing corpora and the scalability challenges of real-world collection.Real collection depends on specialized instruments, calibrated hardware, safety procedures, and domain supervision.
  • Environment Building: The engine builds simulation-ready assets and validates assembled scenes before workflow generation, preventing unreachable placements, unstable contacts, invalid geometry, and unusable rollouts.Scene validation includes ten checks and rejects scenes below a 0–100 quality threshold.
  • Environment Building: The LabAssetLibrary contains 2,947 physics-annotated assets, while workflow configurations support colored liquid proxies and successful source-to-target transfer during pours.Liquid containers declare color and fill fraction, enabling visible liquid behavior in simulation.
  • Workflow Generation: Protocols are represented as templates containing natural-language instructions, named objects, target references, and ordered atomic skills that adapt across robot profiles.The atomic library spans manipulation, instrument interaction, and navigation, while profiles separately encode kinematics, grippers, cameras, and skill overrides.
  • Demonstration Synthesis: After validation, RoboGenesis instantiates workflows across robot platforms and applies controlled randomization to embodiment, lighting, appearance, placement, clutter, and cameras without breaking protocol-behavior correspondence.The system also records checker-specific verdicts, collision events, temporal segments, subgoals, and quality scores for demonstrations.

3. LabVLA Training Recipe

LabVLA uses a two-stage training recipe that first makes the Qwen3-VL-4B-Instruct prefix action aware with FAST tokens, then trains a DiT action expert with flow matching. Knowledge insulation blocks flow gradients from the VLM while preserving token-loss learning, enabling efficient continuous-action inference.

  • Pretraining: LabVLA first jointly trains Qwen3-VL-4B-Instruct on grounded data to produce VQA answers, language subtasks, and discrete FAST action tokens before attaching a continuous action head.The pretraining data sources are Robointer-VQA, AgiBot World Beta, OXE-AugE, and Droid.
  • Embodiment-agnostic training: The embodiment-agnostic batch format pads states and actions to d_max and uses masks to exclude padded dimensions, padded frames, and annotation-only samples from action losses.Dataset schemas configure state and action keys, camera mappings, action dimensions, gripper dimensions, and delta action masks.
  • Flow matching: The second stage attaches a DiT action expert and uses flow matching to map Gaussian noise to clean continuous action chunks.The DiT is an 18-layer model with width 1024, 8 attention heads, and head dimension 128.
  • Inference: 10 Euler steps produce a usable trajectory at sampling time, substantially fewer than the hundreds required by diffusion policies.Posttraining predicts absolute actions, while task-specific LabUtopia finetuning uses delta action targets.
  • Knowledge insulation: Knowledge insulation detaches the VLM prefix from flow gradients while keeping FAST and annotation token losses active, preserving prefix representations for language and visual grounding.Directly cotraining the VLM with flow loss made prefix representations less reliable for downstream attention.

4. Experiments

Experiments evaluate LabVLA on the six-operation LabUtopia benchmark against recent VLA policies spanning sub-1B, 3B, and 4B families. LabVLA achieves the highest average success rate in both ID and OOD settings, with balanced performance across tasks despite greater difficulty in precise liquid pouring.

  • Benchmark: LabUtopia evaluates six laboratory operations under in-distribution and out-of-distribution settings using high-fidelity simulation, procedural scene generation, and hierarchical task benchmarking.The operations are picking up labware, pressing device buttons, opening doors, pouring liquids, heating beakers, and transporting beakers.
  • Baselines: LabVLA is compared with recent VLA policies from sub-1B, 3B, and 4B families under the same LabUtopia protocol.The baselines span flow matching, FAST-tokenized, and synthetic-data-pretrained policies.
  • Quantitative results: 71.1% ID and 70.0% OOD are LabVLA’s average success rates, exceeding the next-best policy 𝜋0 by 7.8 and 6.8 percentage points, respectively.LabVLA leads on Pick Up at 49.2%/48.3% ID/OOD, ties on Open Door at 65.0% ID and 65.8% OOD, and scores 100% on Press Button ID.
  • Quantitative results: 1.1 pp is LabVLA’s ID-to-OOD drop, from 71.1% to 70.0%, while its OOD average remains highest among all baselines.The narrow gap is attributed to visual and spatial invariances from domain randomization in LabEmbodied-Data transferring across scene perturbations.
  • Task difficulty: Pour Liquid is the most demanding task because precise tilting must avoid spills without liquid-level feedback, whereas generous placement tolerances ease multistep tasks.LabVLA is described as the most balanced policy, exceeding 48% on every task except Pour Liquid, while some baselines spike on individual tasks and nearly fail on others.

5. Analysis

The analyses show that LabEmbodied-Data transfers beyond LabVLA to improve an external policy, while LabVLA also transfers from simulation to physical Franka hardware across laboratory tasks and conditions.

  • Transferability to external policies: LabEmbodied-Data lifts fine-tuned X-VLA’s five-task average by +15.0 pp (ID) and +19.3 pp (OOD).The largest gains occur on tasks requiring instrument-specific contact patterns.
  • Transferability to external policies: X-VLA improves most on Heat Beaker (ID: 25.8%→68.3%) and Pour Liquid (OOD: 25.0%→65.0%).These tasks involve instrument-specific contact patterns absent from X-VLA’s original training data.
  • Transferability to external policies: Pick Up is the only task without an in-distribution improvement (27.5%→26.7%), although its OOD performance rises from 27.5% to 31.7%.The results indicate that LabEmbodied-Data provides transferable laboratory supervision beyond the LabVLA architecture.
  • Simulation-to-real transfer: On the four-task real-robot average, DreamZero slightly outperforms LabVLA in cluttered settings: 81.0% vs. 80.0% in domain and 75.5% vs. 74.0% out of domain.The policies are evaluated on a Franka platform across target-position and workspace-clutter conditions.
  • Simulation-to-real transfer: In clean in-domain conditions, DreamZero and LabVLA are within 0.5 pp, at 87.0% vs. 86.5%, while all three policies exceed 70% in most conditions.The four tasks compose 2–4 atomic laboratory skills, and each setting uses 50 rollouts.

6. Related Work

Related VLA work spans mainstream grounding, efficiency, reasoning and memory, and cross-embodiment datasets, but these resources primarily target household or tabletop manipulation. Laboratory systems instead require constrained interaction with instruments, materials, containers, and protocol state, motivating LabVLA’s protocol-conditioned, success-filtered simulation approach.

  • Mainstream VLA training and grounding: Mainstream VLA research generates robot actions from visual observations and language instructions, with CLIPort, BC-Z, RT-1, and RT-2 establishing representative grounding and scaling directions.CLIPort grounds pick-and-place, BC-Z studies language-conditioned imitation for zero-shot task generalization, and RT-1/RT-2 scale Transformer policies with large robot and web-scale data.
  • Efficiency oriented VLA: Efficiency-oriented VLA models reduce model or inference cost through compact architectures, dynamic inference, state-space models, or diffusion policies.SmolVLA and TinyVLA target affordable or data-efficient training, while DeeR-VLA, RoboMamba, and RDT-1B explore dynamic inference, efficient sequence modeling, and diffusion-based action modeling.
  • Reasoning, memory, and spatial VLA: Reasoning, memory, and spatial VLA methods improve generalization by adding reasoning traces, memory, explicit spatial representations, motion reasoning, or instruction tuning.Representative systems include Robotic chain-of-thought, CoT-VLA, CoA-VLA, ThinkAct, FlowVLA, and InstructVLA.
  • Cross-embodiment datasets: Open X-Embodiment, DROID, and BridgeData V2 provide large-scale cross-embodiment supervision for everyday manipulation but exclude laboratory instruments and protocol-conditioned trajectories.LabVLA complements these datasets with success-filtered, protocol-conditioned simulation rather than competing on household and tabletop coverage.
  • Laboratory robotics and benchmarks: Laboratory robotics differs from conventional manipulation because robots must coordinate instruments, containers, materials, and protocol state under constrained planning.Prior work includes specialized self-driving laboratory hardware, simulation-based laboratory benchmarks, and chemistry-oriented planning systems; LabVLA distinguishes itself from prior household/tabletop VLAs and laboratory simulators without paired protocol-conditioned demonstrations.

7. Conclusion

The paper identifies data and embodiment as primary bottlenecks in scientific-laboratory VLA learning and addresses them with LabVLA, RoboGenesis, and LabEmbodied-Data. These artifacts are designed for reuse and extension toward deployment in working laboratories.

  • Conclusion: Data and embodiment are identified as primary bottlenecks alongside model design for scientific-laboratory VLA learning.Executable protocols require shared supervision across scenes, instruments, physical states, and robot morphologies.
  • Conclusion: LabVLA operationalizes this approach by pairing synthesized LabEmbodied-Data from RoboGenesis with FAST action-token pretraining and flow-matching posttraining.
  • Conclusion: RoboGenesis, LabEmbodied-Data, and the LabVLA training recipe are provided as reusable artifacts that can be extended with new instruments, protocols, and robots.The stated goal is to lower the entry cost for laboratory VLA research without rebuilding the underlying infrastructure.
  • Conclusion: Future work will move beyond benchtop evaluation toward deployment in working laboratories with real reagents and instruments under explicit safety constraints.

8. Discussion

The discussion frames laboratory manipulation as stratified across task difficulty and embodied competence rather than adequately summarized by one aggregate score. It also positions RoboGenesis and LabVLA as an early simulated step toward embodied AI for science, with substantial deployment gaps remaining.

  • What the current results support: LabVLA reaches 100%/98.3% ID/OOD on Press Button, while the task is near saturated with most baselines ≥92%.Heat Beaker is solved by GR00T N1.5 at 99.2% and SmolVLA at 96.7%/98.3%, though performance is not uniform across methods.
  • Levels of embodied laboratory competence: Laboratory manipulation is better organized into four competence levels than represented by a single aggregate score.The levels range from Apprentice single-step interactions to Scientist-level capabilities modeled on real laboratory roles.
  • Limitations: Real laboratory deployment remains limited by simulation-heavy validation, a four-task study on one Franka platform, and unaddressed hardware, reagent, safety, contamination, and failure variability.The paper identifies these gaps between the demonstrated setting and real scientific laboratory requirements.
  • From AI for research to embodied AI for science: RoboGenesis and LabVLA provide one practical path from AI for research toward embodied AI for science by grounding laboratory execution in structured workflows and data.The discussion contrasts current AI capabilities in literature, coding, hypothesis design, and experiment planning with execution that still falls to a human operator.

A. Training Hyperparameters

The training setup uses distinct action-target conventions across LabVLA’s stages, with knowledge insulation enabled during the flow-matching posttraining stage.

  • VLM pretraining uses absolute action targets.
  • Finetuning uses delta action targets.
  • KI posttraining is the flow matching stage with knowledge insulation enabled.

B. Training History

LabVLA development exposed training-specific failure modes that training loss alone did not reveal. The final recipe addressed action-dimension gradient scaling, while warmstart checkpoint quality and posttraining data composition mattered more than DiT architecture or learning-rate scheduling.

  • Training Stability: 4×: Averaging flow-matching loss across all 32 padded action dimensions reduced action-expert gradients despite only about 8 dimensions being active for single-arm robots.The final recipe slices the per-element loss to each robot’s active action dimensions.
  • Warmstart Quality: 60% to 86%: Increasing posttraining-data diversity and volume improved TransportBeaker success on a 120-episode evaluation under the same finetuning recipe.The authors attribute this improvement to the action prior learned during posttraining.
  • Training History: Warmstart checkpoint quality and data composition dominated DiT architecture and learning-rate schedule choices during TransportBeaker finetuning.These observations came from training logs and closed-loop evaluation records documenting multidataset VLA training failures.

C. Memory and Compute Optimizations

The section presents implementation-specific memory and throughput optimizations that make training the approximately 5B-parameter Qwen3-VL-4B-plus-DiT stack practical on 80 GB GPUs. These measures include selective checkpointing, fused kernels, attention-mask design, asynchronous batch prefetching, and bounded video-container caching.

  • Selective gradient checkpointing: Selective checkpointing targets the language model, which dominates activation memory, freeing most of the benefit of full checkpointing while limiting recomputation.LabVLA exposes independent checkpointing flags for the vision encoder, language model, and DiT action head, but training enables only the language-model flag.
  • Fused GPU kernels for the VLM backbone: Fused Liger-Kernel operators optimize the Qwen3-VL backbone at RMSNorm, RoPE, SwiGLU, and annotation cross-entropy sites.Fused RMSNorm avoids standard fp32 upcasting overhead, while fused RoPE reduces intermediate sin/cos tensor allocation; a version-gated manual SwiGLU rebind handles a known no-op integration.
  • Attention mask design: The production attention configuration uses decoder-only causal attention with a one-dimensional padding mask over concatenated image, instruction, annotation, and FAST action tokens.This layout is consumed directly by FlashAttention-2, with each token attending to all earlier non-padded positions.
  • Background batch prefetch with a dedicated transfer stream: A producer thread, dedicated CUDA transfer stream, bounded queue, and CUDA events overlap batch loading and host-to-GPU transfer with training.The queue is filled to 80% before the first step.
  • Host memory management for multiadapter video caching: Each dataset adapter maintains an LRU cache of up to 64 open video containers, while persistent workers support the multiadapter streaming mixture.The four-dataset mixture instantiates over 60 adapter shards per DataLoader worker, and the passage reports 60 × 64 = 3,840 as the composed cache bound.

D. Case Study … F. Scene Construction Placement Rules

RoboGenesis constructs diverse laboratory scenes through domain-randomized environments, generated assets, and solver-enforced placement validation. Its asset pipeline uses structured text-to-image prompts, while scene validation scores constraint satisfaction from 0–100 and rejects scenes below threshold.

  • D. Case Study: RoboGenesis generates nine laboratory scenes with randomized layouts, materials, lighting, instrument placement, labware, and background objects.The diversity is designed to reduce visual overfitting and support generalization across laboratory environments.
  • D.1. RoboGenesis Scene Diversity: Scene randomization targets real-laboratory visual variation, including bench surfaces, lighting color temperature, instrument clutter, and wall-mounted items.Examples include posters and safety-related wall objects.
  • E. Asset Generation Prompt Template: The asset-generation prompt fills item, features, material, and viewpoint placeholders from each target object’s textual description.The template supports 3–6 distinguishing visual attributes and physically grounded material descriptions.
  • E. Asset Generation Prompt Template: Assets are requested as isolated, hyperrealistic product photographs with studio lighting, sharp focus, 8K detail, and no text, watermark, label, or background objects.The prompt also excludes multiple items and specifies suitability for 3D reconstruction.
  • F. Scene Construction Placement Rules: The scene construction solver enforces numerical asset-placement constraints during RoboGenesis scene generation.The supplied passage identifies these constraints as values read from the constraint-solver source code.
  • F. Scene Construction Placement Rules: A validation pass computes a 0–100 quality score as the fraction of checks passed and rejects scenes below threshold.This validation filters generated scenes according to constraint satisfaction.
Loading 2606.13578v2…