PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoBio+: Benchmarking Dexterous Manipulation with Dynamic Experimental Phenomena in Realistic Laboratories

Xuanye Wu, Yue Shi, Yongcan Yang, Yiqun Wang, Zhaoqi Wan, Dayu Xia, Jiezhang Cao, Xiangfei Li, Yao Mu, Guangtao Zhai

arXiv:2610.13643v1cs.RO

TL;DR

Biological laboratory automation requires dexterous tool interactions under realistic, changing visual conditions, while existing approaches and benchmarks leave these demands underexplored. AutoBio+ introduces a multi-embodiment simulation benchmark with dynamic phenomena, hybrid rendering, and parallel evaluation. Its evaluations reveal substantial variation in policy performance across tasks, with persistent difficulties in language grounding and contact-rich manipulation.

  • Problem

    Biological laboratory procedures require dexterous tool interactions under realistic visual conditions and dynamic phenomena, but conventional automation relies on task-specific instruments and tightly coupled workflows.

    Source: §I · p. 2

  • Method

    AutoBio+ combines 14 tasks across four embodiments with controllable visual phenomena, hybrid laboratory rendering, and parallel simulation for data generation and policy evaluation.

    Source: §I · p. 2

  • Results

    Policy performance varies substantially across tasks; language grounding remains unreliable, and contact-rich manipulation is a persistent weakness.

    Source: §VI · p. 7

  • Takeaways & Limitations

    Reliable laboratory operations still require task-specific demonstrations and contact-aware control, particularly for precise contact-rich tasks.

    Source: §VI · p. 7

    AutoBio+ does not yet cover the full scope of biological laboratory automation and may not fully reproduce high-fidelity liquid behavior or the full range of scene interactions.

    Source: §VII · p. 8

Abstract

from arXiv · show

Automating biological laboratories requires robots to perform dexterous tool interactions under realistic visual conditions while handling dynamic experimental phenomena, presenting significant challenges for robotic manipulation. To enable systematic evaluation under these conditions, we introduce AutoBio+, a simulation benchmark and framework for dexterous manipulation in realistic biological laboratory environments. AutoBio+ comprises 14 tasks across 4 robot embodiments, spanning functional tool use, precision manipulation, language-conditioned visual grounding, and instrument operation. We introduce controllable dynamic experimental phenomena, including flames, vapor, and bubbles, to capture the evolving visual conditions of laboratory procedures. To further enhance environmental realism, we develop a hybrid rendering pipeline that integrates interactive simulated foreground objects with Gaussian-splatting backgrounds reconstructed from 6 real laboratory scenes. For efficient large-scale data generation and evaluation, we further build a parallel simulation framework on top of mjlab, proposing Parallel Finite-State Machine (PFSM) and integrating batched rendering with decoupled trajectory replay for demonstration generation and model-independent closed-loop policy evaluation. Our evaluation reveals substantial variation in policy performance across tasks, exposing distinct strengths and weaknesses of current manipulation policies. Real-world experiments further demonstrate a reduced sim-to-real gap, supporting the validity of AutoBio+ for realistic manipulation evaluation. Supplementary materials are available at https://autobio-plus.github.io/AutoBioPlus.

I. INTRODUCTION

Biological laboratory procedures demand dexterous, adaptable tool use, while existing automation and benchmarks leave dynamic laboratory conditions and fine-grained laboratory manipulation underexplored. AutoBio+ addresses this gap with a multi-embodiment task suite and simulation framework.

I. INTRODUCTION

  • AutoBio+ evaluates 14 laboratory tasks across four robot embodiments, covering tool use, precision manipulation, language-conditioned visual grounding, and instrument operation.These tasks broaden scientific robotics evaluation beyond limited task suites and simplified hand control.Source: §I · p. 2, §II · p. 3
  • Controllable flames, vapor, and bubbles add process-dependent visual changes that policies must handle during laboratory procedures.Existing manipulation benchmarks underrepresent these temporally varying conditions.Source: §I · p. 2
  • Hybrid rendering combines interactive simulated objects with Gaussian-splatting backgrounds reconstructed from real laboratories, preserving physical interaction while improving visual realism.The background scenes are reconstructed from six real laboratories.Source: §I · p. 2
  • Built on mjlab, the framework combines PFSM, parallel rendering, and decoupled trajectory replay for demonstration generation and model-independent closed-loop evaluation.This supports efficient large-scale data generation and benchmarking.Source: §I · p. 2

II. RELATED WORK

  • AutoBio+ targets laboratory procedures that general-purpose manipulation benchmarks and gripper-focused scientific benchmarks do not fully represent.Dexterous benchmarks study coordinated hand control, but laboratory-specific tool use and dynamic experimental phenomena remain less explored.Source: §II · p. 3

III. AUTOBIO+ SIMULATION

AutoBio+ Simulation extends mjlab with dynamic visual effects, hybrid rendering, and parallel simulation. Its particle system approximates laboratory phenomena with lightweight rendering-only primitives rather than full physics.

III. AUTOBIO+ SIMULATION

  • AutoBio+ Simulation extends mjlab with enhanced rendering, dynamic experimental phenomena, and system-level parallelization while retaining the underlying physics engine.The framework comprises a particle visual-effects system, hybrid rendering, and parallel simulation.Source: §III · p. 3

A. Particle System for Dynamic Experimental Phenomena

  • Each rendering-only particle stores position, color, opacity, and size, with state decoupled from scene physics.This lightweight, parallelizable design favors perceptual fidelity over expensive full-physics modeling of phenomena such as vapor and changing flame intensity.Source: A. Particle System for Dynamic… · p. 3
  • Particle motion combines deterministic drift with random perturbation, while particle size and opacity can change over its lifetime.Drift can encode buoyancy and upward motion; stochastic perturbations produce flicker and irregular motion, and age-dependent opacity decay is specified by a time constant.Source: A. Particle System for Dynamic… · p. 3
  • Flames, vapor, and bubbles use distinct particle behaviors, including burner-dependent flame color, expanding vapor clouds, and rising vortex-like bubble paths.These effects approximate their visual dynamics while remaining decoupled from the physics simulation.Source: A. Particle System for Dynamic… · pp. 3–4

B. Hybrid Rendering for Realistic Laboratory Environments

The hybrid renderer combines simulator-rendered interactive foreground objects with Gaussian-splatting backgrounds to provide realistic laboratory context without sacrificing physical interaction.

B. Hybrid Rendering for Realistic Laboratory Environments

  • Hybrid rendering composites Gaussian-splatting laboratory backgrounds with simulator-rendered foreground objects, retaining interactive physical control.The Gaussian assets come from six laboratory scenes captured with a lightweight spatial scanner.Source: B. Hybrid Rendering for Realistic… · p. 4
  • The pipeline preserves interactive simulated objects while supplying photorealistic laboratory backgrounds.It avoids replacing the simulator renderer entirely.Source: B. Hybrid Rendering for Realistic… · p. 4
  • The compositing uses mjlab’s object-level foreground mask to combine the simulator image with the rendered Gaussian scene.The scene Gaussians are rendered from the camera pose before the foreground and background are combined.Source: B. Hybrid Rendering for Realistic… · p. 4

C. Parallel Simulation

AutoBio+ supports parallel data generation and evaluation, with rendering and visual effects designed to scale across environments. Its task design emphasizes functional tool use, precision, language grounding, and instrument operation.

C. Parallel Simulation

  • Batched Gaussian rendering processes multiple environments simultaneously, while particle updates run independently without global synchronization.The framework also predefines textures in MJCF and indexes them during simulation to support texture variability with parallel execution.Source: C. Parallel Simulation · p. 4
  • 15.57 s for Gaussian-augmented rendering was approximately 70.14× as fast as AutoBio’s 1092.02 s, with 11.61% overhead over plain MuJoCo Warp’s 13.95 s.The benchmark used 339 frames from two cameras at 640 × 480 resolution.Source: C. Parallel Simulation · p. 4

A. Design Principles for Tasks

  • The task design requires using laboratory tools for their intended functions and precise, coordinated finger motion during contact-rich manipulation.These principles distinguish laboratory manipulation from generic object rearrangement.Source: A. Design Principles for Tasks · p. 4
  • Tasks require grounding language instructions in scene entities and operating instruments through articulated controls and physical manipulation.A spatial instruction can specify a tube-rack location by row and column.Source: A. Design Principles for Tasks · p. 4

B. Dataset Generation

AutoBio+ generates grasp candidates, filters them for physical feasibility, and manually validates them before using a parallel finite-state policy to collect demonstrations. It also decouples physics rollout from rendering and includes tasks and evolving dynamic phenomena.

Grasp Synthesis

  • Grasp candidates from Dexonomy and Lightning Grasp are converted into embodiment-specific pre-grasp and grasp targets for trajectory planning.The candidates are generated from object geometry and hand kinematics to cover a wider variety of grasp types.Source: B. Dataset Generation · pp. 4–5

Dynamic Experimental Phenomena

Grasp Synthesis

  • Parallel simulation filters grasps that are unreachable, collide with the environment, or fail to maintain stable object contact during motion.Retained candidates are kinematically feasible and dynamically stable under the contact dynamics used for demonstration collection.Source: B. Dataset Generation · p. 5
  • Manual inspection removes physically feasible but unnatural grasps, retaining human-like configurations for the grasp dataset.Examples of rejected poses include self-intersecting fingers and awkward configurations that can hinder task execution.Source: B. Dataset Generation · p. 5

PFSM Policy for Data Collection

  • The parallel finite-state policy decomposes tasks into semantic stages and waypoints, with independent environments able to advance asynchronously.Each environment maintains its own stage and waypoint indices; state updates determine completion and advance the corresponding stage.Source: B. Dataset Generation · pp. 5–6
  • Physics rollout and image rendering are decoupled: only successful trajectories are replayed in batches to render aligned multi-view observations.Disabling camera rendering during rollout avoids rendering during trajectory search while retaining frame-level alignment between control and visual observations.Source: B. Dataset Generation · p. 6

A. Simulation Benchmark Evaluation

AutoBio+ evaluation found higher real-world scores with Gaussian-splatting backgrounds than with standard MuJoCo backgrounds in a few-shot setting. Simulation results also show uneven policy performance, with persistent challenges in language grounding, dexterous contact, and precise contact-rich manipulation.

A. Simulation Benchmark Evaluation

  • Four policies were compared on 12 representative tasks using shared task-specific training data, identical domain randomization, and 300 rollouts per task.The task-wise success rates are reported as mean ± SEM over three seeds, with 100 rollouts per seed.Source: A. Simulation Benchmark Evaluation · p. 6, Table II · p. 7

VI. DISCUSSION

  • Language-conditioned grounding success remains low and variable across policies, and none of the evaluated tasks is solved reliably.The tasks require matching language-specified spatial or symbolic targets to scenes and combining identification with spatial reasoning and precise motion control.Source: §VI · p. 7

B. Real-World Validation

  • 66% versus 17% normalized mean score was achieved with Gaussian backgrounds versus MuJoCo backgrounds across 20 real-robot trials.Each training set contained 10 real and 90 simulated demonstrations; the real-world task used a UR5e with an XHand and wrist-mounted and third-person cameras.Source: B. Real-World Validation · p. 6, Figure 5 · p. 7, Figure 4 · p. 7

VI. DISCUSSION

  • GR00T-N1.7 achieves 100% success on Close Centrifuge Lid, but this benefit does not consistently extend to contact-rich pipette, tweezer, and tube-cap manipulation requiring precise alignment and finger coordination.Source: §VI · p. 7
  • Policies perform worse on precise contact-rich tasks than on nonprehensile interactions, highlighting contact-aware control and fine-grained visuomotor coordination as needs for reliable laboratory manipulation.Examples include tweezer manipulation, tightening tube caps, attaching pipette tips, loading centrifuges, and transferring tubes.Source: §VI · p. 7

VII. LIMITATIONS AND FUTURE WORK

AutoBio+ combines a dexterous laboratory benchmark with parallel simulation and hybrid rendering, while its evaluations show varied policy performance and improved few-shot transfer in real-world experiments. The authors identify limits in task coverage and dynamic-scene fidelity.

VII. LIMITATIONS AND FUTURE WORK

  • The benchmark does not yet evaluate the full scope of biological laboratory automation, and future asset and task synthesis could reduce manual effort and broaden evaluation coverage.The current Gaussian-background compositing and particle-based system may not fully reproduce high-fidelity liquid behavior or the full range of scene interactions.Source: §VII · p. 8

VIII. CONCLUSION

  • Evaluations reveal substantial cross-task variation and persistent limitations in contact-rich interaction, language–spatial grounding, and fine-grained dexterous manipulation.Real-world experiments also show that Gaussian-splatting backgrounds reduce the sim-to-real gap and improve few-shot transfer.Source: §VIII · p. 8
  • AutoBio+ covers 14 tasks across four robot embodiments and four capability categories, with controllable flames, vapor, and bubbles representing process-dependent visual changes.Its hybrid rendering combines real-laboratory 3D Gaussian Splatting backgrounds, and PFSM supports efficient demonstration generation and policy evaluation.Source: §VIII · p. 8

APPENDIX

The appendix describes demonstrations as time-aligned observation–action trajectories, with observations combining camera images, proprioception, and language instructions. It also specifies task-specific camera views and action components.

A. Dataset Structure

  • Each demonstration is stored as a time-aligned trajectory of observations and actions.At each timestep, observations contain multi-view camera images, robot proprioception, and a language instruction.Source: A. Dataset Structure · p. 10
  • Camera superscripts identify the view, and Tube Transfer (XHand) includes only the front and wrist1 views to match the real-robot setup.The language instruction is associated with the trajectory.Source: A. Dataset Structure · p. 10
  • The proprioceptive vector contains arm joint positions and hand or gripper joint positions.The end effector is an articulated hand in dexterous setups and a gripper in simpler setups.Source: A. Dataset Structure · p. 10
  • Actions contain end-effector position, orientation as a unit quaternion, and hand- or gripper-joint positions.For dual-arm tasks, the action concatenates commands for both arms.Source: A. Dataset Structure · p. 10

B. Visualization and Language Instruction of AutoBio+ Tasks

The appendix specifies how AutoBio+ presents task scenes and instructions, randomizes episode starts, and supports policy evaluation through simulator checks and decoupled inference. It also describes porting its thread mechanism to mjlab for parallel simulation.

B. Visualization and Language Instruction of AutoBio+ Tasks

C. Randomization Settings of AutoBio+ Tasks

  • At episode reset, scene randomization varies object poses, discrete slot assignments, and applicable visual appearance, while robot randomization perturbs initial joint configurations within task-specific ranges.Table IV reports the sampling rules and reset ranges for each task.Source: C. Randomization Settings of AutoBio+… · p. 10

D. Evaluation Infrastructure

  • The evaluation infrastructure combines domain randomization, task-specific success checks, and model-independent policy communication.Success checks determine task completion from simulator state.Source: D. Evaluation Infrastructure · p. 10
  • A ZeroMQ observation–action interface decouples the simulator from the policy runtime, which can run in an independent Python environment.The simulator sends visual observations, proprioception, and language prompts, then receives action chunks.Source: D. Evaluation Infrastructure · p. 10
  • Policies can be evaluated from a separate process, machine, or software environment without integration into the simulator's dependency stack.This follows from the external inference process exchanging observations and action chunks with the simulator.Source: D. Evaluation Infrastructure · p. 10
  • AutoBio+ ports AutoBio's thread mechanism to mjlab, enabling large-scale parallel simulation from CPU to GPU while preserving its behavior.Source: D. Evaluation Infrastructure · p. 10

E. Training Details

The appendix reports training settings for simulated tasks and the real-world Transfer Tube (XHand) experiment. Figure 6 shows raw training-loss curves for the simulated tasks and that real-world experiment.

E. Training Details

  • Simulated-task models are fine-tuned for 30,000 optimization steps on one NVIDIA H200 GPU, with π0.5 using batch size 32.Source: E. Training Details · p. 10
  • For real-world Transfer Tube (XHand), π0.5 is trained on 100 demonstrations for 30,000 steps with batch size 32 and a 30-step action horizon.Training uses a peak learning rate of 5 × 10^-5, 3,333-step warmup, AdamW, weight decay 10^-10, and gradient-norm clipping at 1.0.Source: E. Training Details · p. 10
  • Figure 6 plots raw training losses for 12 simulated tasks and the real-world Transfer Tube (XHand) experiment.The orange panel marks real-world training, while blue panels mark simulated-task training.Source: E. Training Details · p. 10, Figure 6 · p. 13
Loading 2610.13643v1…