Source-linked AI summary
Labference: Benchmarking the Reasoning-Action Connection in Robotic Laboratories
Shitong Liu, Jingwei Zhang, Sheng Lian, Xinyu Zhuang, Kaiwen Guo, Quan Liu, Chunhai Zhao, Jiaxin He, Fan Yang, Manli Tao, Tingyu Yuan, Jinqiao Wang, Hongwei Shi, Chaomin Guo, Xiao-jing Liu, Xiangyu Zhu, Zhuang Xiong, Feichan Yang, Chaoyang Zhao, Zhiyong Liu
TL;DR
Existing evaluations leave unresolved how outcome-dependent choices translate into physical execution, despite the need for robots to adapt operations to actual results. Labference tests this connection through controlled evidence variations, separate choice and execution measures, and matched support continuations. Initial weighing results show target-selection errors in tested VLA policies and execution and efficiency difficulties for GPT-driven control.
Problem
Aggregate completion and separate reasoning or execution metrics leave unresolved how physical outcomes guide subsequent choices and how those choices translate into actions.
Source: §1 · p. 1Method
Labference uses controlled evidence variations and matched checkpoint continuations to measure operation selection, execution, completion, and the effects of targeted support.
Source: §1 · pp. 2, 4, §2.4 · p. 6Results
The tested VLA policies selected the required weight in 20% and 18% of their 50-episode evaluations and completed 16% and 8%, while the GPT-driven system selected correctly in all ten trials and completed eight.
Source: §10 · p. 17Takeaways & Limitations
The findings motivate interfaces that preserve task requirements through action generation while delivering feedback quickly enough to sustain a procedure.
Source: §10 · p. 18Current evidence is limited to aggregate outcomes from randomized easy-weighing episodes and qualitative execution cases; matched controlled pairs and event-level records remain planned.
Source: §9.7 · p. 17
Abstract
from arXiv · showhide
Realizing Embodied AI for Science requires robots to translate experimental goals into reliable physical procedures, reducing hazardous and repetitive human work while generating evidence for scientific discovery. Although language-based planning and learned manipulation provide complementary capabilities, their integration requires robots to adapt subsequent operations to actual execution outcomes. For example, an unsuccessful placement changes what must be done next, even when the experimental goal remains unchanged. Existing benchmarks assess planning, manipulation, and laboratory workflows through task completion and reasoning or execution metrics; however, aggregate performance leaves the connection between outcome-dependent choices and physical execution unresolved. To address this evaluation gap, we introduce Labference, a benchmark for evaluating how observed physical outcomes guide subsequent choices and how those choices translate into robotic laboratory actions. Its evaluation protocol is designed to test outcome-dependent behavioral adaptation through controlled evidence variations, while independently measuring operation selection, physical execution, and task completion. Matched Oracle-assisted continuations from shared checkpoints then assess which forms of support improve subsequent choices and task completion. Initial weighing evaluations reveal target-selection errors in the tested VLA policies despite successful object transfers, whereas GPT-driven control achieves correct selection but encounters execution failures and substantial computation costs. Together, these findings highlight the need to connect task-appropriate reasoning with reliable, timely action. By making this connection measurable and diagnosable, Labference supports targeted improvements toward autonomous experimentation for Embodied AI for Science.
1 Introduction
Labference frames robotic laboratory autonomy as a reasoning–action problem: robots must preserve experimental requirements through physical operations and adapt choices to observed outcomes. It introduces controlled evidence variations and matched support continuations to diagnose selection, execution, and completion separately.
1 Introduction
- Autonomous laboratories aim to reduce hazardous, repetitive work and use experimental results to guide subsequent investigations, but robots must preserve requirements and adapt to unexpected outcomes.Laboratory procedures require sustained precision around chemicals, heated vessels, and moving equipment.Source: §1 · p. 1
- Retrieval after rack relocation requires remembering a sample’s local slot and locating it in the rack’s current pose, because grasping a neighboring tube can succeed physically yet violate procedure.The evaluation distinguishes target association, localization, and grasp execution.Source: §1 · p. 2
- State-dependent reasoning applies task rules to current evidence and relevant interaction history to select an operation, object, destination, or waiting or stopping condition.Physical confirmation establishes whether the resulting action fulfilled the requirement.Source: §1 · p. 2
- Labference tests whether controlled changes in physical evidence change the required choice, then records choice, execution, completion, correction, and recovery separately.Tasks define legal next operations and progress-confirming physical events, accommodating multiple valid operation orders.Source: §1 · pp. 2, 4
- Matched Oracle continuations alter a registered input or interface output from common checkpoints to compare subsequent choices and physical results with a base branch.Support can target state recognition, operation selection, target localization, or execution.Source: §1 · pp. 2–3
- The tested VLA policies sometimes transfer an incorrect standard weight, while the GPT-driven system selects the required weight in all ten trials but completes eight.Drops affect completion, and prolonged deliberation limits practical deployment.Source: §1 · p. 3
2 Related Work
Related work establishes complementary capabilities in laboratory automation, feedback-dependent planning, process understanding, and policy infrastructure. Labference focuses on whether actual outcomes determine appropriate next operations and how support affects their physical realization.
2.1 Autonomous laboratories and scientific manipulation
- Autonomous-laboratory systems connect experimental planning to physical action through instrument programs, coordinated workflows, or visually grounded manipulation, while Labference studies outcome-dependent operational choices.Laboratory benchmarks also address physical, procedural, dexterity, and safety requirements.Source: §2.1 · p. 4
2.2 Feedback-dependent planning and diagnostic evaluation
- Language-to-action methods ground instructions in skills or spatial constraints, and feedback benchmarks test plan adaptation or intermediate interfaces; Labference links measured outcomes to subsequent choices and physical execution.It records choice, execution, and completion separately from common checkpoints.Source: §2.2 · p. 4
2.3 History, process understanding, and access to evidence
- History-focused benchmarks assess memory, process understanding, or observation conditions; Labference couples a robot’s realized placement with rack relocation and records recall, retrieval choice, and execution separately.The evidence and history supplied define the memory condition.Source: §2.3 · p. 5
2.4 Policy systems, infrastructure, and the comparison boundary
- Policy systems and evaluation infrastructure provide distinct roles; Labference uses declared interfaces to test evidence-dependent behavior and support while retaining each system’s observation, action, and budget conditions.Its measurements connect adaptation, checkpoint interface correctness, and downstream response to targeted support.Source: §2.4 · p. 6
3 Evaluating State-Dependent Operations
This section defines appropriate next operations from confirmed task state and tests whether policies change choices when relevant evidence changes. Task contracts and controlled comparisons separate outcome-dependent selection from physical execution.
3.1 From action sequences to outcome-dependent choices
- The reference legal-operation set Uτ(zt, g) depends on task state and goal, while physical outcomes and device updates advance state.Several operations may be legal, allowing alternative orders and recovery.Source: §3.1 · p. 6
3.2 Task contracts
- A TaskContract links decision-relevant state and authorized evidence to legal operations, physical confirmation, waiting and terminal conditions, and diagnostic interfaces.Events retain time, object identity, measured target, and attempt identifiers so repeated contacts or requests cannot complete an obligation twice.Source: §3.2 · p. 6
3.3 Shared task-design principles
- Task design separates unfinished obligations, evidence needed to score choices, and physical confirmation of completion.Initialized objects establish context, while policy credit begins with subsequent interaction; spatial loading, weighing, and remembered placement instantiate the shared outcome-to-choice relation.Source: §3.3 · p. 6
3.4 Controlled changes to the evidence
- Matched instances hold instructions and nuisance conditions fixed while changing evidence that requires a different next operation.Answer-changing pairs test selection, failed-versus-successful placements test recovery, and geometric-equivariance pairs test realizing the same local choice after relocation.Source: §3.4 · pp. 6–7
4 Laboratory Task Families
The benchmark instantiates outcome-dependent choices in centrifuge loading, feedback weighing, and placement recall. Each task distinguishes the choice required by actual outcomes from whether the robot physically completes it.
4 Laboratory Task Families
- Three task families test spatial, numerical, and historical dependencies by making realized outcomes determine the next appropriate operation.They cover centrifuge loading, feedback weighing, and retrieval of a previously placed sample.Source: §4 · p. 7
4.1 Centrifuge loading: acting on actual occupancy
- Centrifuge loading requires the second tube’s destination to oppose the first tube’s actual stable slot, while failed or unfinished placement leaves that operation pending.Choice and physical confirmation are measured separately; completion requires both tubes stable in opposite slots.Source: §4.1 · p. 7
4.2 Feedback weighing: revising a numerical plan
- Feedback weighing selects additions or removals from the target, measured reading, and standards actually supported on the pan.A missed weight leaves its obligation outstanding; completion also requires the target reading, exact standard set, release, low motion, and continuous stability.Source: §4.2 · pp. 7–8
4.3 Placement recall: using an earlier observed result
- Placement recall uses the tube’s actual released rack slot as the retrieval reference, including after occlusion and rack movement.Same-cap and distinct-cap conditions vary available appearance cues; choice, retrieval execution, and completion are separately recorded.Source: §4.3 · p. 8
- Autonomous episodes measure self-managed interaction histories, while balanced checkpoints compare systems given common placement histories across all twelve slots.The populations answer complementary questions and use a fixed denominator for the checkpoints.Source: §4.3 · p. 8
5 Online Choice and Execution Evaluation
The online protocol links policy-visible evidence to choices, physical actions, and task completion while recording information boundaries and timing. It scores behavior at distinct decision, execution, and full-task levels.
5.1 Policy interfaces and information boundaries
- The tested system includes its model, memory, tools, observation channels, and action adapter, while policy observations, Oracle context, and evaluator state remain separate data views.The harness records the evidence delivered to the policy; actual outcomes produce fresh observations and subsequent choices, with evaluation and demonstration recording as separate outputs.Source: §5.1 · pp. 8–9, Figure 2 · p. 9
5.2 A timed closed loop
- A single runtime step owner links observations, policy actions, applied commands, physical measurements, and camera delivery times.Waiting, inference latency, and tool calls are recorded; simulated-time stability and wall-clock responsiveness are distinct measurements.Source: §5.2 · p. 9
5.3 Event-defined decision opportunities
- A decision opportunity opens when relevant new evidence reaches the policy, and the first identifiable behavioral choice is frozen separately from later revisions.A correct target followed by a dropped object earns choice credit but counts as execution failure.Source: §5.3 · p. 10
5.4 Results at three levels
- Evaluation separates legal-operation choice, physical subgoal execution from a common start, and full-task completion under protocol and safety conditions.On-policy results include reach rates and opportunity counts, while shared checkpoints support cross-system comparisons of the same decision.Source: §5.4 · p. 10
6 Diagnosing Capability Bottlenecks
The section diagnoses system capabilities at shared checkpoints and tests support through matched continuations. It separates interface accuracy, support delivery and agreement, and downstream benefit to connect measurements with potential intervention sites.
6.1 What can the system do at a common state?
- Verified checkpoints fix physical state and authorized evidence, enabling tests of state recognition, operation choice, geometry, and execution.A correct subgoal and standard geometry isolate physical execution, while history prefixes preserve differences in the evidence available to stateful policies.Source: §6.1 · p. 10
6.2 Which support improves subsequent behavior?
- Oracle support either adds information while the system produces its output or replaces an output at an existing interface, with original outputs retained for direct scoring when accessible.Support can target semantic facts, decisions, geometry, or execution through compatible interfaces.Source: §6.2 · pp. 10–11
6.3 Intervention timing and delivery
- Each intervention is bound to an opportunity, evidence version, and application site, and delivery, recommendation agreement, and outcome benefit are measured separately.The validator checks content scope and records delivery and outputs; unsupported interfaces receive compatibility status, while transport faults are validity events.Source: §6.3 · p. 11
6.4 Matched online continuations
- Matched base and supported branches share a verified checkpoint and system conditions, linking support to recommendation, choice, and physical outcome.Accurate readout with improved selection under Decision support indicates operation selection as a useful intervention site; accurate selection with improved completion under Skill support points to physical realization.Source: §6.4 · p. 11
7 Comparison and Reporting Principles
The comparison framework separates operation choice, physical execution, and verified completion, with fixed populations and explicit coverage to make failures interpretable. It also tests adaptation across answer-changing pairs and records compatibility, incomplete evidence, and uncertainty conditions.
7.1 Choice correctness, identification, and reach
- Choice metrics use fixed verified, evidence-delivered checkpoints to distinguish no identifiable choice by the deadline from an incorrect choice after evidence delivery.Complete rollouts separately report reach, correctness conditional on reach, and correctness over all runs.Source: §7.1 · pp. 11–12
7.2 Execution, completion, and recovery
- Physical execution is measured on shared verified starts with the same correct operations and geometric references, while completion requires sustained physical predicates and valid protocol and safety checks.Clean completion requires correct identified choices without recorded operation failures or protocol violations; recovered completion requires documented error resolution and valid task completion.Source: §7.2 · p. 12
7.3 State-changing and geometric pairs
- State-pair performance tests adaptation by requiring legal choices on both members of answer-changing pairs, while geometric controls preserve local target identity under rigid scene transforms.The paired legal sets are disjoint at the scored choice, and localization and execution are evaluated in the transformed scene.Source: §7.3 · p. 12
7.4 Compatibility, incomplete data, and uncertainty
- Compatibility is checked before evaluation, missing runs and service faults reduce coverage, and uncertainty is resampled over episodes or matched pairs with rollout decisions kept together.An agent built around a VLA is compared at the system level because it changes tools, context, and computation.Source: §7.4 · p. 12
8 Reference Implementation and Reproducibility
The reference implementation uses shared runtime components and task-owned measurements to connect physical state, observations, and actions. It specifies force-derived weighing feedback, controlled task and environment conditions, persistent records, and staged validation for reproducible comparisons.
8.1 Shared runtime and task-owned measurements
- A shared Isaac Sim and PhysX runtime supports task-owned measurements, legal-operation generators, and scorers for centrifuge loading, feedback weighing, and placement recall.The centrifuge bridge checks insertion geometry, release, and support, while the rack separates guide handoff tolerances from measured seating success.Source: §8.1 · pp. 12–13
8.2 Force-derived numerical feedback
- Weighing feedback comes from a calibrated tray-support force measurement, filtered into a seven-segment readout, with completion requiring a stable in-range reading and supported, released contents.The reference configuration uses a 0.25 s filter time constant, 0.1 g display quantum, and a continuous 3 s terminal window.Source: §8.2 · p. 13
8.3 Generation, history, and environment transitions
- Task generation separates state and layout seeds, while placement recall varies distractor appearance and evidence conditions and scores first slot choice separately from physical recovery.The hard rack condition reconstructs tubes after an occluded rack movement while preserving measured local pose and logical identity.Source: §8.3 · p. 13
8.4 Policy adapters and persistent records
- Policy adapters enter comparisons after capability preflight and online handoff, and persistent event, observation, score, and demonstration records retain identities and execution details for replay audit.Replay audit reconstructs choices and physical confirmations with the online scorer and checks evidence-version consistency.Source: §8.4 · pp. 13–14
8.5 Validation and comparison records
- Validation checks task rules and scoring denominators before matching episodes, proceeding from geometry and sensor calibration through interface handoff, complete execution, and matched model evaluation.Protocol tests include stale evidence, interrupted stability, unsupported interventions, and incomplete pairs.Source: §8.5 · p. 14
9 Experiments
Initial weighing experiments separate operation selection from physical execution: VLA policies often transfer an inappropriate mass, while GPT-driven control selects correctly but sometimes fails to complete transfers. The evaluations also report substantial computation time and outline matched-checkpoint tests and broader task extensions.
9.1 Feedback weighing as an initial test
- Feedback weighing pairs a stable scale reading and target mass with a choice among four standards, followed by grasping, transfer, release, and verification.The initial study examines whether manipulation serves the mass requirement and whether actions are fast enough for practical use.Source: §9.1 · p. 14
9.3 Choosing a weight and executing its transfer
- VLA policies select the correct weight in 20% and 18% of episodes and complete 16% and 8%, respectively; GPT selects correctly in all ten trials and completes eight.The reported rates distinguish selection from completion across the evaluated systems.Source: Table 3 · p. 14, §9.3 · p. 14, §9.2 · p. 14
9.2 Evaluated systems and deployment conditions
- The evaluations include task-specific π0.5 and GR00T-N1.5 policies, while the GPT-driven study uses camera and robot-state observations in ten varied-initialization trials.The VLA studies use 50 episodes each; Harness-VLA remains pending.Source: §9.2 · p. 14
9.3 Choosing a weight and executing its transfer
- In failed VLA cases, policies transfer an inappropriate standard despite successful grasping and placement; observations also report repeating one standard as requirements change.This indicates that executable transfer behavior does not ensure selection tracks the numerical requirement.Source: §9.3 · pp. 14–15
9.4 From a correct choice to a stable grasp
- GPT trials select the required standard, but some transfers fail when the grasp allows the weight to slip or fall, prompting regrasp attempts.The evaluation distinguishes physical realization and recovery from the correctness of the selected weight.Source: §9.4 · p. 15, Figure 3 · p. 15
9.5 Runtime and deployment efficiency
- Mean episode runtimes are 135.60 s for π0.5 and 86.81 s for GR00T-N1.5; GPT reports about 30 minutes of model computation and 50 seconds of task execution per trial.In the synchronous simulator, physics pauses during inference, unlike a continuously evolving process with a finite response window.Source: §9.5 · pp. 15, 17
9.6 Interpreting the reasoning–action connection
- The VLA and GPT cases expose different breakdowns: inappropriate mass selection versus grasp execution, motivating support tests for choice, realization, and timely continuation.Matched checkpoints compare support from common physical starts; execution-specific comparisons fix the selected subgoal.Source: §9.6 · p. 17
9.7 Limitations and continued evaluation
- Current evidence is limited to aggregate randomized easy-weighing outcomes and qualitative execution cases; planned tests add matched state changes, per-call timing, and broader tasks.Extensions include centrifuge loading and placement recall, with targeted improvements tested on held-out instances.Source: §9.7 · p. 17
10 Conclusion
Labference evaluates whether observed outcomes guide appropriate laboratory choices and whether those choices lead to successful physical execution. Initial weighing trials expose distinct selection and execution failures, while the benchmark’s shared-checkpoint protocol supports targeted evaluation of assistance.
10 Conclusion
- Labference separates selection, execution, and completion measurements across spatial, numerical, and history-dependent choices.Shared checkpoints and matched Oracle continuations test which information or execution support helps restore progress.Source: §10 · p. 17
- In feedback weighing, VLA policies selected required weights in 20% and 18% of their 50-episode evaluations and completed 16% and 8%, while GPT selected correctly in all ten trials and completed eight.Reported GPT failures included drops and regrasp attempts; its computation took approximately 30 minutes per trial versus roughly 50 seconds of execution.Source: §10 · pp. 17–18
- The initial weighing results motivate interfaces that preserve task requirements through action generation while providing feedback quickly enough to sustain a procedure.Source: §10 · p. 18
- Measuring whether observed outcomes lead to appropriate choices and successful execution provides a basis for improving the connection from experimental intent to action and from realized results to the next operation.Source: §10 · p. 18
A Laboratory Task Catalog
The catalog spans laboratory workflows and reasoning-in-action tasks, with descriptions organized around operations, evidence, and intended outcomes.
A Laboratory Task Catalog
- The catalog contains twelve laboratory workflows (L01–L12) and twelve reasoning-in-action tasks (R01–R12), described by operation, relevant evidence, and intended outcome.Its three task families develop spatial loading, quantitative feedback, and placement history.Source: A Laboratory Task Catalog · p. 21
A.1 Sample handling and preparation
The catalog covers sample preparation, equipment use, readout-driven choices, and occupancy or coordinate reasoning. Across these tasks, the intended outcome specifies both the appropriate operation and its physical result.
A.1 Sample handling and preparation
- Sample-handling tasks cover vial routing, liquid transfer and mixing, recipe dispensing, and ordered balance loading, with outcomes defined by correct assignments and completed placements or transfers.Source: §A.1 · p. 21
A.2 Equipment use and workspace restoration
- Equipment tasks include centrifuge and NMR loading, device operation, sequential vial placement, and bench reset, with identity, destination, and sequence shaping completion.Source: §A.2 · p. 22
A.3 Readouts, device states, and relational choices
- Readout-driven tasks map mass, weight, temperature, RPM, or device state to a choice such as selecting, adjusting, waiting, or opening.Source: §A.3 · p. 23
A.4 Occupancy, local coordinates, and access conditions
- Occupancy and coordinate tasks require choosing available symmetric rotor slots, mapping numbered or grid locations to sockets, or selecting an action from lid state.Source: §A.4 · p. 24
B Physical-Robot Pilot Examples
The physical-robot pilot examples document recorded manipulation outcomes, while the quantitative results reported elsewhere concern simulated easy-weighing evaluations.
B Physical-Robot Pilot Examples
- Figure 9 shows keyframes from supplied successful and failed physical-robot recordings, with each row following a recording in temporal order.The pilot outcome labels follow experimental records; Table 3 reports separate simulated easy-weighing results.Source: B Physical-Robot Pilot Examples · p. 25, Figure 9 · p. 25