PaperlayerView exact revision on arXiv

Source-linked AI summary

SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents

Maokai Qin, Chuan Qin, Qi Zhang, Dianyu Liu, Zirui Liu, Hongting Niu, Yuanchun Zhou, Hengshu Zhu

arXiv:2609.30971v1cs.AI

TL;DR

Scientific protocols describe procedures at a high level, while embodied evaluation requires explicit task representations and objective process-level success criteria; existing construction is commonly manual or component-specific. SciHorizon-eLab grounds protocols in simulator environments, generates executable manipulation programs and independent step-level success specifications, and certifies tasks through simulation. The strongest policy averages 49.7% task success across ten representative certified tasks, and coordination evaluations show zero PVR but below-17% final task success for both tasks.

  • Problem

    Scientific protocols describe procedures at a high level, while embodied evaluation requires explicit task representations and objective process-level success criteria; existing construction is commonly manual or component-specific.

    Source: §1 · p. 3

  • Method

    SciHorizon-eLab grounds protocols in simulator environments, generates executable manipulation programs and independent step-level success specifications, and certifies tasks through simulation.

    Source: §1 · p. 3

  • Results

    The strongest policy averages 49.7% task success across ten representative certified tasks, and coordination evaluations show zero PVR but below-17% final task success for both tasks.

    Source: §1 · p. 3, §5.2.2 · p. 11

  • Takeaways & Limitations

    SciVLABench offers 300 certified tasks supporting reproducible expert demonstrations, human-agent collaboration, and ordered step-level evaluation.

    Source: §1 · p. 3

    The remaining 19% of candidates fall outside current automated recovery scope and require lightweight human intervention, averaging 1–2 minutes per task.

    Source: §5.3 · p. 12

Abstract

from arXiv · show

Embodied agents offer a promising route to automating scientific experimentation, yet their progress is constrained by the lack of reliable and systematic evaluation environments. Existing simulation-based laboratory benchmarks rely heavily on manual task engineering, making it challenging to systematically compile diverse scientific protocols into executable and verifiable embodied tasks at scale. To address this challenge, we introduce SciHorizon-eLab, an agentic protocol-to-task compiler that formulates scientific embodied task construction as a compilation problem. Given a natural-language protocol of scientific experiments, SciHorizon-eLab progressively compiles laboratory protocols into semantic-preserving embodied tasks through semantic grounding, executable task synthesis, and multi-stage simulation-based certification. The system generates semantically grounded environments, executable manipulation programs, and step-level success specifications, while enabling reproducible generation of expert demonstrations and execution traces. Using this pipeline, we further construct \BenchName, a ready-to-use benchmark comprising 300 certified tasks across diverse laboratory operations. It supports HIL task execution, reproducible expert-demonstration generation, and ordered step-level evaluation. Across representative tasks, the strongest policy attains an average success rate of only 49.7%, with further evaluations revealing pronounced weaknesses in human and embodied agent coordination. We publicly release the code, benchmark data, and evaluation toolkit at https://github.com/SciHorizon-elab/SciHorizon-elab.

1 Introduction

Scientific embodied-agent evaluation needs protocol-grounded tasks that preserve experimental intent while making actions and success criteria executable and verifiable. SciHorizon-eLab compiles protocols into certified tasks, and SciVLABench provides 300 tasks for reproducible demonstrations and evaluation; representative policies still struggle.

1 Introduction

  • Scientific protocols describe procedures at a high level, whereas embodied benchmarks need explicit entities, actions, states, constraints, and measurable success conditions.Existing benchmarks commonly rely on manual or user-configured task structures, and component-level automation does not provide an end-to-end protocol-grounded construction and certification process.Source: §1 · p. 3
  • SciHorizon-eLab compiles natural-language protocols through semantic-preserving environment grounding, executable manipulation and independent step-level success specifications, then simulation-based certification.The pipeline validates structural correctness, physical executability, and ordered success conditions before task instantiation.Source: §1 · p. 3
  • SciVLABench comprises 300 certified tasks built from 51 protocol-grounded scenarios, with environments, executable reference programs, and ordered success specifications for reproducible demonstrations and evaluation.It also includes 15 human-agent collaborative tasks and randomized instances that preserve task semantics.Source: §1 · pp. 3–4
  • 49.7% average task success is achieved by the strongest policy across ten representative certified tasks, while coordination and held-out object-binding evaluations expose further limitations.The experiments also audit the compiler’s first-pass reliability, recoverability, and remaining manual effort.Source: §1 · p. 3

2 Related Work

Prior laboratory and manipulation benchmarks provide useful simulated environments, while automated construction systems address parts of task creation. The cited work leaves protocol-grounded task compilation with independent process evaluation and certification as an open gap.

2.1 Scientific Embodied Agents

  • Scientific laboratory platforms such as Chemistry3D, LabUtopia, and Pipette provide interactive assets and evaluation, but their tasks are primarily manually specified or user-configured.These simulation environments reduce the cost and safety risks of physical experimentation.Source: §2.1 · p. 4

2.2 Embodied Agent Benchmarking

  • General manipulation benchmarks evaluate standardized skills, geometry generalization, long-horizon composition, and reasoning, while laboratory benchmarks focus on scientific environments.Real-robot evaluation is costly, difficult to standardize, and constrained by hardware and safety requirements.Source: §2.2 · p. 4
  • Automated task-construction systems address components such as task proposal and program synthesis, but do not systematically derive task definitions and evaluation from scientific protocols or certify semantic preservation.SciHorizon-eLab compiles protocols into grounded environments, executable procedures, and independent step-level criteria, followed by simulation certification.Source: §2.2 · p. 4

3 SciHorizon-eLab

SciHorizon-eLab compiles natural-language scientific protocols into embodied tasks that can be instantiated, executed, and evaluated. Its pipeline grounds protocol steps and environments, generates aligned execution and success specifications, then certifies tasks through simulation.

3.1 Framework Overview

  • The three-stage pipeline grounds protocols in environments, generates executable task specifications with independent success criteria, then certifies and instantiates tasks in simulation.The stages are Semantic-Preserving Environment Compilation, Executable Task Specification Generation, and Simulation Certification and Task Instantiation.Source: §3.1 · p. 5

3.2 Semantic-Preserving Environment Compilation

  • Grounding preserves protocol order and explicit operations in Q, while E records entities, asset bindings, relations, and initial-state constraints for simulation.Entity references remain consistent, assets must support required interactions, and simulator-specific additions enable execution without changing protocol-level entities, operations, or outcomes.Source: §3.2 · pp. 5–6

3.3.1 Executable Procedure Generation.

  • The execution planner generates P over the full procedure, tracking intermediate states and adding robot-specific actions such as grasping or repositioning without changing experimental order or outcomes.A deterministic validator checks skill registration, parameters, entity bindings, and preservation of Q's order.Source: §3.3.1 · p. 6

3.3.2 Success-Condition Generation.

  • An independently prompted planner derives step-level verification conditions from the grounded protocol, covering observable states and temporal constraints without access to execution program P.A deterministic validator checks that each grounded step has a supported condition and consistent entity references; both outputs share the same procedure and environment.Source: §3.3.2 · pp. 6–7

3.4.1 Task Compilation and Physical Execution.

  • The certification stage compiles tasks into simulator-native classes, validates and registers them, then executes P in MuJoCo to produce a trace of states, skill status, waypoints, and observations.Registration or execution failures exclude a candidate from verification and are recorded for inspection or refinement.Source: §3.4.1 · p. 7, §3.4 · p. 7

3.4.2 Complementary Task Verification.

  • Certification checks ordered step conditions against simulator states and uses visual review of step-aligned observations to assess procedural semantics not fully captured by predicates.Both channels are required so certified tasks are executable and faithful to the intended procedure.Source: §3.4.2 · p. 7

3.4.3 Certification and Task Instantiation.

  • A task is certified only when compilation and registration, execution, predicate verification, and visual-semantic review all succeed.The certification result is defined as Γ(τ) = 1.Source: §3.4.3 · p. 7
  • Certified tasks generate instances by sampling admissible poses, workspace layouts, initial states, and visual conditions while preserving protocol-level roles, programs, and success conditions.The resulting randomized instances become task resources for the benchmark.Source: §3.4.3 · pp. 7–8

4 SciVLABench

SciVLABench packages certified scientific laboratory tasks for direct execution and evaluation. It spans protocol-grounded operations and supports randomized task instances, reproducible demonstrations, ordered procedural assessment, and coordination evaluation.

4 SciVLABench

  • SciVLABench provides 300 certified autonomous tasks and 15 human-agent coordination tasks, retaining grounded environments, expert programs, and ordered success specifications for direct instantiation and evaluation.Researchers can generate expert demonstrations and evaluate policies without rerunning protocol-to-task compilation.Source: §4 · p. 8

4.1 Task Sources and Benchmark Scope

  • The benchmark curates 51 protocol-grounded scenarios across five operation families, while explicitly treating this taxonomy as operational scope rather than an exhaustive map of experimentation.The families are liquid handling and transfer, mixing and agitation, solid handling and weighing, thermal control and incubation, and apparatus and workspace interaction.Source: §4.1 · p. 8
  • Using 30 registered atomic skills, the scenarios yield 300 certified tasks; a separate 15-task subset tests waiting for external events, recognizing them, and safely resuming procedures.Admissible variations in poses and workspace configurations preserve each task's operation semantics, executable procedure, and success specification.Source: §4.1 · p. 8

4.2 Released Resources and Evaluation Protocols

  • Each task package supports reproducible demonstrations and traces, while ordered specifications assess complete or partial procedural completion and coordination safety separately from final completion.Evaluation protocols cover same-task randomization, held-out geometrically distinct object bindings, and randomized timing of external interventions.Source: §4.2 · pp. 8–9

5 Experiments

The experiments assess policy performance, procedural completion, coordination, transfer, and compiler reliability through benchmark evaluations and a controlled audit.

5.1.1 Policies and Demonstrations.

  • Three policies—π0.5, ACT, and Diffusion Policy—are evaluated using demonstrations generated by each task’s certified expert program, over 30 randomized episodes per task.The episodes use independently sampled initial conditions.Source: §5.1.1 · p. 9

5.1.2 Policy Inputs and Training.

  • Policies receive four RGB views and robot proprioception, and predict 7D end-effector commands; privileged simulator states are reserved for demonstration generation and outcome evaluation.ACT and DP use 100K optimization steps; π0.5 is fine-tuned for 100K steps.Source: §5.1.2 · p. 9

5.1.3 Evaluation Metrics.

  • Task SR requires the complete ordered success specification, while SSR measures the fraction of intended procedure completed before failure.The evaluation uses 30 episodes, and step-wise success counts only after procedural prerequisites are satisfied.Source: §5.1.3 · pp. 9–10

5.2.1 Policy Comparison and Procedural Diagnosis

  • ACT achieves the highest average SR, while π0.5 achieves the highest average SSR; on Transport and Mix, success is at most 3.3% despite SSR reaching 42.5%.The difference between complete success and step-wise completion indicates that policies often perform early operations before failing later.Source: Table 2 · p. 10, §5.2.1 · p. 10

5.2 Benchmark Evaluation

  • Evaluation covers policy performance, coordination with external interventions, and transfer to held-out object bindings, beyond aggregate task success alone.The benchmark is evaluated on ten certified tasks spanning five operation families.Source: §5.2 · p. 10, §5.2.1 · p. 10

5.2.2 Human-Agent Coordination

  • Both coordination policies have zero PVR, yet task success stays below 17% and valid post-trigger responses take a mean 6.8–8.3 seconds.The results distinguish safe waiting from intervention detection, appropriate response, and subsequent task completion.Source: Table 3 · p. 11, §5.2.2 · p. 11

5.2.3 Generalization to Held-Out Object Bindings

  • All three policies degrade on held-out object bindings, with larger degradation for transport than lifting; average held-out SR is 76.7% for π0.5, 45.0% for ACT, and 23.3% for DP.The transfer tests preserve the operation and success semantics while changing object geometry and affordances.Source: §5.2.3 · pp. 11–12, Table 4 · p. 11

5.3 Compilation and Certification Audit

The controlled audit measures task-construction reliability within the registered asset and operation libraries, finding that automated certification recovers many but not all candidates.

5.3 Compilation and Certification Audit

  • 58% of generated tasks pass certification checks on the first attempt; automatic recovery raises the fully automated certification rate to 81%.The recovery mechanisms add 23% of candidates through simulator rerunning and visual-state correction.Source: §5.3 · p. 12
  • Certification uses diagnostic feedback to repair transient execution failures and inconsistent initial configurations before benchmark admission.Visual-state correction addresses mismatches between generated scenes and textual task specifications, while predicate and visual-semantic checks verify procedural outcomes.Source: §5.3 · p. 12
  • The remaining 19% of candidates fall outside current automated recovery scope, primarily because task-specific spatial and interaction configurations are not covered; lightweight human intervention corrects these cases in 1–2 minutes per task on average.Source: §5.3 · p. 12

6 Conclusion

The conclusion presents SciHorizon-eLab as a protocol-to-task compiler and SciVLABench as a certified benchmark for evaluating scientific embodied agents.

6 Conclusion

  • SciHorizon-eLab combines semantic grounding, executable task synthesis, and multi-stage simulation certification to compile natural-language protocols into executable, verifiable tasks.The system is framed as a compilation approach to embodied task construction.Source: §6 · p. 12
  • SciVLABench supports human-in-the-loop interaction, reproducible expert demonstrations, and fine-grained step-level evaluation across laboratory scenarios.Source: §6 · p. 12
  • Experiments find substantial challenges for current embodied policies, particularly in long-horizon manipulation and human-agent coordination.The conclusion characterizes the framework as a scalable paradigm for converting scientific knowledge into executable embodied experiences.Source: §6 · p. 12

A.1 Primary-Family Assignment Principles

SciVLABench assigns each scenario to one primary operation family according to the physical operation that realizes its scientific objective. The five families distinguish liquid transfer, mixing, solid handling, thermal control, and apparatus or workspace interaction.

Primary-family assignment

  • Scenario assignment follows the physical operation that primarily realizes the scientific objective, not disciplinary context, laboratoryware, or secondary capabilities.For multi-stage scenarios, the primary family is selected by the operation that directly leads to the intended experimental outcome.Source: §A.1 · p. 13

Operation families

  • Liquid handling and transfer covers operations such as aspiration, dispensing, pouring, and transfer, while mixing and agitation centers on homogenizing or agitating materials.A procedure belongs to mixing when achieving a mixed or homogenized state, rather than merely transferring material, is its intended outcome.Source: §A.1 · p. 13
  • Solid handling and weighing covers manipulating and quantitatively handling solid samples; thermal control and incubation covers establishing, maintaining, monitoring, or safely ending thermal conditions.Thermal procedures may involve heating, incubation, drying, temperature monitoring, and post-heating handling.Source: §A.1 · p. 13
  • Apparatus and workspace interaction covers equipment and workspace assembly, configuration, placement, storage, recovery, and organization, including safety-related recovery when apparatus state is central.The benchmark assigns each scenario to one primary family based on its dominant embodied operation; the family taxonomy characterizes scope rather than exhaustively categorizing scientific experimentation.Source: §A.1 · p. 13

Family composition

  • Table S1 summarizes the family-level composition of the 51 protocol-grounded source scenarios.The complete scenario-to-family mapping is provided in Table S2.Source: Table S1 · p. 13, §A.1 · p. 13

A.2 Scenario-to-Family Mapping

The scenario-to-family mapping assigns each of SciVLABench’s 51 source scenarios to one primary operation family. Analytical, sensing, safety, and human-intervention properties remain orthogonal metadata rather than additional primary families.

Family abbreviations

  • LHT, MA, SHW, TCI, and AWI abbreviate Liquid Handling and Transfer, Mixing and Agitation, Solid Handling and Weighing, Thermal Control and Incubation, and Apparatus and Workspace Interaction..Source: §A.2 · p. 13

Scenario-to-family mapping

  • Table S2 presents primary-family assignments for the 51 source scenarios, with source scenario and primary family as the mapping fields.The continued table includes examples such as equipment-door and drawer closing and collision-aware glassware transport, both assigned to AWI.Source: Table S2 · p. 14, §A.2 · p. 15

Secondary metadata

  • Sensing, analytical testing, safety-critical execution, and human intervention are retained as secondary task metadata rather than defining additional primary operation families.Examples include thermometer-based temperature measurement assigned to TCI, pH-paper sampling assigned to LHT, and spill cleanup assigned to AWI.Source: §A.2 · p. 15

B Registered Atomic Skill Library

SciHorizon-eLab uses a registered library of 30 atomic manipulation skills to compile grounded protocol steps into robot-executable programs. A planning agent composes these skills, and a deterministic validator checks that programs use registered skills with valid arguments.

B Registered Atomic Skill Library

  • SciHorizon-eLab compiles grounded protocol steps using a registered library of 30 atomic manipulation skills.The skill-planning agent can compose multiple skills to realize one protocol step, and a deterministic validator checks skill registration and argument validity.Source: B Registered Atomic Skill Library · p. 15
  • Each skill is a reusable simulator-level operation parameterized by grounded entities, target poses, spatial offsets, or task-specific execution settings.Source: B Registered Atomic Skill Library · p. 15
  • The skill-planning agent can combine multiple atomic skills for one protocol step, while a deterministic validator requires registered skills and complete, valid arguments.Source: B Registered Atomic Skill Library · p. 15
  • The callable simulator interfaces include robot reset and gripper control, object manipulation, waiting, and liquid handling such as aspiration and dispensing.Table S3 reports the skill names using the exact callable identifiers in the simulator execution API.Source: Table S3 · p. 15, B Registered Atomic Skill Library · pp. 15–16

Related papers

Loading 2609.30971v1…