PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboChemGym: A Protocol-Driven Generative Simulation Framework for Long-Horizon Chemical Manipulation

Chenxi Li, Haiyuan Wan, Rui Li, Jingyuan Li, Sha Zhang, Bohan Feng, Jianbao Cao, Zhangrui Zhao, Di Hu, Wangmeng Zuo, Shixiang Tang, Minting Pan, Dongzhan Zhou

arXiv:2610.02708v1cs.RO

TL;DR

Real-world laboratory training data are costly and risky to collect, and existing simulation often misses chemical protocols' long-horizon demands. RoboChemGym generates protocol-aligned demonstrations through iterative action-and-scene refinement and establishes a hierarchical benchmark. Its benchmark covers 20 tasks, including workflows with up to 11 atomic actions; the system remains limited to the Franka Panda and simulation-based validation rather than fully autonomous real-world execution.

  • Problem

    Real-world laboratory training data are costly and risky to collect, while existing simulation methods largely target short-horizon, loosely structured tasks rather than protocol-constrained chemical workflows.

    Source: Introduction · p. 1

  • Method

    RoboChemGym converts textual chemical protocols into executable scenes and actions, iteratively refining action plans and layouts to generate demonstrations and a hierarchical benchmark.

    Source: Introduction · pp. 1–2

  • Results

    The protocol-derived benchmark covers 20 tasks, including 3 ultra-long-horizon tasks and workflows with up to 11 atomic actions.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The benchmark supports atomic- and procedural-level performance analysis, including error accumulation in long-horizon executions and standardized comparisons across methods.

    Source: Introduction · p. 2

    Evaluation relies extensively on simulation, and RoboChemGym currently supports only the Franka Panda.

    Source: Conclusion & Limitations · p. 8

Abstract

from arXiv · show

Wet-lab experimentation serves as the gold standard for hypothesis verification in scientific discovery; yet it is inherently labor-intensive, costly, and safety-critical. Embodied agents hold the promise of automating these tedious workflows, but their development is hindered by the scarcity of real-world training data. While simulation offers a scalable alternative for producing demonstrations, current methods primarily target relatively short-horizon tasks with loosely structured interactions, failing to meet the strict procedural constraints and fine-grained manipulation demands of chemical experiments. To bridge this gap, we introduce \textbf{RoboChemGym}, a framework that autonomously generates high-fidelity manipulation demonstrations aligned with real-world experiment protocols, featuring a \textit{self-improving task synthesis} mechanism to iteratively refine task execution and scene configurations, enabling the reliable generation of expert trajectories for complex, multi-object protocols exceeding 10 interaction steps. Furthermore, we introduce a hierarchical benchmark that systematically assesses performance across varying granularities, spanning from atomic operations to full-cycle experimental workflows. RoboChemGym sets a scalable paradigm for the automated data synthesis and capability evaluation of embodied agents in intricate chemical tasks, serving as a critical stepping stone toward fully intelligent laboratories.

Abstract

RoboChemGym generates protocol-aligned chemical manipulation demonstrations by refining actions and scenes, and introduces a benchmark spanning atomic operations to long-horizon workflows.

Introduction

  • Real-world lab data are costly and risky to collect, while existing simulation approaches do not meet the protocol and long-horizon demands of chemical workflows.RoboChemGym addresses this gap with protocol-aligned simulation and a hierarchical benchmark.Source: Introduction · pp. 1–2

Related Work

Protocol-Driven Data Generation

  • RoboChemGym turns textual chemical protocols into executable tasks through protocol parsing, joint scene-and-action refinement, and lab-constrained data variation.The pipeline produces diverse demonstrations consistent with laboratory conventions and physical constraints.Source: Protocol-Driven Data Generation · p. 3

Generative Protocol Instantiation

  • A language model extracts scene descriptions and action sequences, which scene and planning engines combine with laboratory assets and parameterized atomic-action primitives.The action library includes Pick, Place, Pour, Press, Open, Close, Shake, and Stir.Source: Generative Protocol Instantiation · p. 3

Self-Improving Task Synthesis

  • Continuous monitoring feeds execution failures into inner action and outer scene optimization loops, while action and scene memories preserve corrective experience for later iterations.The inner loop adjusts action parameters; persistent failures can trigger scene-layout changes.Source: Self-Improving Task Synthesis · pp. 3–4

Lab-Centric Data Scaling

  • Lab-centric scaling varies object positions, contextual distractors, workbench materials, and lighting while maintaining physically plausible experimental scenes.Object-position randomization is constrained to preserve safety and task feasibility.Source: Lab-Centric Data Scaling · p. 5

Protocol-Aligned Hierarchical Benchmark

The benchmark organizes protocol-derived chemical tasks by horizon and object complexity, then evaluates performance at both atomic-action and full-task levels.

Protocol-Aligned Hierarchical Benchmark

Hierarchical Task Setup

  • Task complexity is categorized by action horizon and object complexity: Level 1 has 3 atomic actions and one target; Level 2 has 7 actions, multiple targets, and two to three objects; Level 3 has 11 actions and more than three objects.Source: Hierarchical Task Setup · p. 5

Multi-Granularity Long-Horizon Evaluation

  • Evaluation sets goal states and tolerance thresholds for atomic actions, while full-task success requires every action in the sequence to succeed.This captures the path dependency of experiments and supports analysis of error accumulation across long workflows.Source: Multi-Granularity Long-Horizon… · p. 5

Experiments

Experiments assess trajectory generation on chemical protocols, manipulation-policy performance under clean and randomized data settings, and sim-to-real transfer. Across these evaluations, generated trajectories show stable success across difficulty levels, randomized data benefits policy performance, and synthetic-data augmentation improves real-world results.

Evaluation of Automated Task Generation

  • Level-3 generation success rises from 33% in early iterations to around 48%, then stabilizes at 52% in Clean and approximately 47% in Random settings.Removing action memory cuts clean Level-3 success from 52.3% to 27.4%; complex failures chiefly involve multi-object layouts and interaction constraints.Source: Figure 3 · p. 6, Evaluation of Automated Task Generation · p. 6
  • 89.2%, 73.4%, and 52.3% are the pipeline’s clean-setting success rates across Levels 1–3; randomized-setting rates are 85.6%, 70.3%, and 47.5%.The pipeline is evaluated on 20 chemical experiment protocols.Source: Evaluation of Automated Task Generation · pp. 5–6

Evaluation of Various Manipulation Policies

  • π0 achieves 18.2% (Clean) and 28.4% (Random) full-sequence success on Level-2, while ACT and DP fall to 5.2% and 3.8% on Level-2 and near zero on Level-3.π0+random reaches 1.7% full-sequence success on Level-3; training improves most individual-action success rates despite low overall success.Source: Evaluation of Various Manipulation… · pp. 6–7, Table 2 · p. 7, Table 3 · p. 7
  • Randomized-data models consistently outperform their clean-data counterparts, with π0 rising from 55.4% to 68.5% on Level-1 and from 18.2% to 28.4% on Level-2.The reported performance gap widens with task complexity, and π0+random has larger step-wise gains in later Level-2 and Level-3 stages.Source: Evaluation of Various Manipulation… · p. 7

Evaluation on Sim-to-Real Performance

  • Adding 1,000 domain-randomized synthetic trajectories to 10 real demonstrations increases average real-world success by 23.3% across evaluation configurations.Synthetic-only zero-shot policies also show moderate improvements, reaching 5.3%, 4.5%, 5.2%, and 6.0% across the configurations.Source: Evaluation on Sim-to-Real Performance · p. 7

Conclusion & Limitations

RoboChemGym is proposed as an automated embodied manipulation framework for chemical experiments. The authors identify limits in real-world execution validation and robot-embodiment coverage.

  • RoboChemGym is proposed as an automated embodied manipulation framework for chemical experiment tasks.Source: Conclusion & Limitations · p. 7
  • Table 4 reports real-world success rates across task configurations distinguished by training position visibility and clutter.Source: Table 4 · p. 8
  • Real-world validation remains limited to trajectory replay rather than fully autonomous execution, and the framework currently supports only the Franka Panda.Source: Conclusion & Limitations · p. 8

Computational Cost Analysis

The analysis estimates LLM overhead from empirical API-call logs and reports call patterns, task-level estimates, and latency considerations. Costs remain modest because long-context calls are used sparsely alongside localized refinements.

Computational Cost Analysis

Logging Setup and Task-Level Aggregation

Empirical Call Patterns

  • Short calls use about 1337 input tokens and 1360–1530 output tokens, medium calls use 2400–2600 input and 2200–2550 output tokens, while infrequent heavy calls use 18k–69k input and about 3k–8k output tokens.Short calls typically handle localized action fixes, medium calls include richer execution context, and heavy calls support sparse global reasoning.Source: Empirical Call Patterns · p. 11

Per-Task Token Footprint and Cost

Latency Considerations

  • Per-call TTFW typically ranges from 15–40 seconds for short and medium calls, with occasional larger values for heavy calls; task-level latency is not reported as a simple call-time sum.Task-level wall-clock time also depends on generation duration, environment rollouts, and server-side queueing.Source: Latency Considerations · p. 11

Discussion

  • Even for Level 3 tasks, estimated monetary cost is on the order of one US dollar per task, with sparse long-context calls supporting graceful scaling.The framework uses localized, structured refinements rather than repeatedly invoking global reasoning over entire protocols.Source: Discussion · p. 12

Scene Construction Details

Initial scenes are assembled from chemical assets and laboratory environments, then asset placement is optimized within a robot-reachable region. The layout objective balances spatial separation, center alignment, and visibility while enforcing a minimum distance.

  • Initial laboratory scenes draw on over 140 high-fidelity chemical assets and more than 10 laboratory environments, with retrieved assets instantiated to match extracted scene specifications.Source: Scene Construction Details · p. 12
  • Asset placement is constrained to an elliptical planar region estimated from sampled robot kinematics and must satisfy collision, distance, visibility, and operational-center constraints.Source: Scene Construction Details · p. 12
  • The weighted placement objective balances spatial separation, center alignment, and visibility, with a 15 cm minimum distance between objects.The stated weights are 0.75, 0.6, and 0.35, respectively.Source: Scene Construction Details · p. 12

Data-Scaling and Visual Variations

The lab-centric data-scaling strategy varies visual and spatial conditions while preserving operational feasibility and experimental validity. These variations expand training conditions and are intended to improve robustness and sim-to-real generalization.

Protocols and Task Grouping

The benchmark groups protocol-derived tasks by action count and manipulation complexity, from short single-target sequences to 11-action workflows involving more than four targets. Protocol examples illustrate the steps and equipment involved at each level.

Protocols and Task Grouping

  • The protocols are organized into task levels to assess embodied-agent performance across short- and ultra-long-horizon tasks.The grouping increases task complexity across the levels.Source: Protocols and Task Grouping · p. 13

Level 1: Short-Horizon Tasks with Single Target

Level 2: Long-Horizon Tasks with Multiple Targets

Level 3: Ultra-Long-Horizon Tasks

Additional Sim-to-Real Results

Physical-robot trajectory replay evaluates whether generated action sequences execute successfully under reconstructed randomized scenes. Reset actions improved replay success, with larger gains on the more difficult tasks.

Physical-robot evaluation

  • The replay study sampled 300 trajectories from 20 protocols across three difficulty levels and reconstructed each randomized object placement before replaying the full action sequence.Success required completing the sequence without failures such as object displacement or execution interruption.Source: Additional Sim-to-Real Results · p. 14

Reset-based correction

  • Figure 6 shows reset actions raised average replay success by 9.4 percentage points, from 69.3% to 78.7%, across all tasks.The reset returns the arm to an initial configuration well within joint limits after each pick-and-place pair.Source: Figure 6 · p. 15, Additional Sim-to-Real Results · p. 15

Physical-robot evaluation

Reset-based correction

  • Reset actions improved replay success at every level, with gains of 1.6 points at Level 1, 10.5 at Level 2, and 15.9 at Level 3.The authors report that the reset mechanism mitigates error accumulation and supports stable execution in long-horizon replay.Source: Figure 6 · p. 15, Additional Sim-to-Real Results · p. 15

Policy Training and Inference Details

The study compares three imitation-learning policy families using a shared observation and joint-space control interface. Training recipes and temporal context differ by policy.

Policy Architectures

  • ACT generates temporally chunked actions autoregressively, Diffusion Policy denoises future action sequences, and π0 combines a PaliGemma backbone with a flow-matching action expert.The π0 design is intended for extended temporal horizons and compositional tasks.Source: Policy Architectures · pp. 15–16

Model Inputs and Outputs

  • All policies receive workspace images and robot proprioception and output continuous joint-space commands; ACT and Diffusion Policy use front and overhead views, while π0 uses wrist and front views.Inputs are 256 × 256 RGB images and absolute joint positions, and the predicted target joint configuration is executed by a closed-loop controller.Source: Model Inputs and Outputs · p. 16

Training Details

  • ACT and Diffusion Policy train for 200 epochs on one RTX 4090, while π0 trains for 15,000 updates on four H200 GPUs, with different frame histories and action chunk lengths.All use a fixed learning rate of 1 × 10−4; ACT and π0 use one observation frame, Diffusion Policy uses three, and chunk lengths are 60 for ACT and Diffusion Policy versus 8 for π0.Source: Training Details · p. 16

Prompts Used for Data Generation and Evaluation

The appendix specifies prompts for converting laboratory protocols into structured task representations, generating controllers, and iteratively adjusting controller parameters and scene layouts. Their constraints emphasize consistency with the system’s assets, interfaces, and execution requirements.

Prompt for Scene and Action Extraction

  • Protocol-to-action parsing extracts an equipment inventory and ordered atomic actions from natural-language laboratory procedures.A closed asset vocabulary maps equipment to predefined containers and devices; solid and liquid reagents receive representative containers, and some controller actions require prior grasping.Source: Prompt for Scene and Action Extraction · pp. 16–17
  • The extraction prompt requires a predefined asset mapping, device-operation rules, generic placement targets, and consistency between inventory and action sequence.Its output must contain two marked blocks, and the resulting program should be complete and executable under the stated constraints.Source: Prompt for Scene and Action Extraction · pp. 16–17

Prompt for Controller-Class Code Generation

  • Controller-code generation targets task-specific classes that support both action execution and data collection within the project framework.The prompt specifies reset() and step(state), predefined primitive controllers, staged execution, guarded data collection, and safe handling of missing state entries.Source: Prompt for Controller-Class Code… · pp. 17–18
  • The controller prompt requires exact object naming, maintained grasp state, default orientation with task-dependent adjustments, and Python-only output.When enabled, collection mode caches relevant step data and stores final state information before termination.Source: Prompt for Controller-Class Code… · p. 18

Prompt for Controller-Parameter Optimization

  • Controller-parameter optimization uses execution logs to diagnose likely failures and make targeted parameter changes without altering controller logic.It favors minimal, evidence-based edits and can reuse adjustments from similar past cases while preserving safety and compatibility.Source: Prompt for Controller-Parameter… · pp. 18–19

Prompt for Layout Optimization

  • Layout optimization addresses manipulation failures by adjusting object placement, particularly when gripper access is the issue.It fixes object height at 0.775 and constrains placement by reachability, non-overlap, reduced occlusion, and spatial relationships; it changes one object’s position at a time.Source: Prompt for Layout Optimization · pp. 19–20

Structured Monitoring Feedback and Memory Representations

A rollout trace shows how monitoring diagnoses execution errors, guides local refinement, and escalates to scene refinement when local corrections stall. Monitor outputs also become structured memory for future task synthesis.

Monitor Output: Failure Detection and Attribution

  • For the pick failure, the monitor reports a gripper-to-target distance of about 0.37 m against a success threshold below 0.1 m, indicating a substantial grasp-region deviation.The trace attributes the failure to the end-effector not reaching the intended grasp region.Source: Monitor Output: Failure Detection and… · p. 21

Action Refinement Output: Execution-Level Correction

  • For a grasp failure attributed to misalignment, action refinement updates end-effector orientation while leaving other factors unchanged.The update is intended to improve alignment without changing the overall task structure.Source: Action Refinement Output:… · p. 21
  • After repeated action refinements show no sufficient progress, the monitor treats persistently high deviation as scene-level infeasibility and escalates.The trace shows three refinement iterations before escalation.Source: Action Refinement Output:… · p. 21
  • Scene refinement addresses geometric constraints through object-layout and grasp-configuration changes, after which the post-check reports constraints satisfied.Listed updates include adjusting object position, modifying orientation, and updating the grasp anchor.Source: Action Refinement Output:… · p. 22

Structured Monitoring Feedback and Memory Representations

  • Structured records of diagnoses and scene updates preserve pose deltas, anchor offsets, and violated constraints for reuse in similar future tasks.These records can let future instantiations bypass repeated trial-and-error refinement and initialize scenes in a feasible region.Source: Action Refinement Output:… · p. 22

Related papers

Loading 2610.02708v1…