PaperlayerView exact revision on arXiv

Source-linked AI summary

LIBERO-Industry: Benchmarking Robotic Manipulation under Industrial Process Constraints

Ming Shin Cindy Sung, Duan Leijiahao, Haoxiang Wang, Boning Tan, Liangna Fu, Chao Tan, Yafu Li, Yulong Chen, Chaoqun Wan, Zhen Huang, Shuxian Liang, Xian-Sheng Hua

arXiv:2610.16782v1cs.AI

TL;DR

Industrial manipulation requires adapting to changing products and production conditions while meeting process constraints that existing evaluation frameworks do not systematically capture for generalist policies. LIBERO-Industry translates these constraints into controllable task conditions and measurable success criteria across three transfer scenarios. Its evaluations show progressive degradation as constraints accumulate and higher observed target-transfer completion for the LIBERO-Industry co-training condition than for original-LIBERO.

  • Problem

    Generalist robot policy evaluation does not systematically capture industrial process constraints such as changing products, moving material flows, finite execution windows, and tight geometric tolerances.

    Source: §1 · pp. 1–2

  • Method

    LIBERO-Industry extends LIBERO with three loading, unloading, and transfer suites that operationalize industrial constraints as controllable task conditions and measurable success criteria.

    Source: §1 · p. 2, §5 · p. 10

  • Results

    Performance degrades across the cumulative industrial stress progression, and LIBERO-Industry co-training achieves higher observed target-transfer completion than original-LIBERO under a shared fine-tuning and evaluation protocol.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    The benchmark provides a LIBERO-based setting with controlled variations in task configuration and operating conditions for evaluating generalist policies under industrial process constraints.

    Source: §5 · p. 10

    Finite opportunity windows abstract heterogeneous factory needs; future evaluation should use production-calibrated settings and test repeated arrivals, recovery events, and long-horizon throughput.

    Source: §5 · p. 11

Abstract

from arXiv · show

Industrial manipulation in flexible manufacturing, e.g., high-mix, low-volume (HMLV) electronics assembly, requires robot policies to adapt to changing products and production conditions. Such settings impose industrial process constraints, including product variants, moving material flows, finite execution windows, and tight geometric tolerances. However, existing evaluation frameworks do not systematically capture these constraints when assessing generalist robot policies. To address this gap, we introduce LIBERO-Industry, a novel benchmark built on the widely used robot learning benchmark LIBERO, that translates the aforementioned industrial constraints into controllable task conditions and measurable success criteria. Specifically, LIBERO-Industry comprises three classic task scenarios: Static-to-Dynamic (S2D) loading, Dynamic-to-Static (D2S) unloading, and Dynamic-to-Dynamic (D2D) transfer. The suites contain 216 tasks in total that systematically vary object geometry, conveyor speed, interaction-window duration, and geometric acceptance region size. Extensive experiments across five generalist robot policies (including vision-language-action models and world-action models) show that performance progressively degrades as industrial constraints are introduced and combined. Under a shared fine-tuning and evaluation protocol, the LIBERO-Industry co-training condition achieved higher observed target-transfer completion than the original-LIBERO condition.

1 Introduction

Industrial process conditions make flexible manufacturing difficult to capture with task-level adaptability alone, yet existing benchmarks do not systematically evaluate generalist policies under those conditions. LIBERO-Industry operationalizes them as controllable tasks and evaluates policies across three industrial scenarios.

1 Introduction

  • Industrial manipulation combines product variants, moving material flows, finite execution windows, and tight geometric tolerances, which conventional automation accommodates through reprogramming or workcell changes.These requirements motivate evaluating industrial process constraints alongside task-level adaptability.Source: §1 · pp. 1–2
  • LIBERO-Industry extends LIBERO with Static-to-Dynamic loading, Dynamic-to-Static unloading, and Dynamic-to-Dynamic transfer scenarios.The suites distinguish whether workpiece sources and destinations are stationary or moving.Source: §1 · p. 2
  • The study evaluates five policies using task success, throughput, and stage-wise completion, and tests physical D2S performance on an AUBO-i5H arm.The real-world trials compare LIBERO-Industry and LIBERO fine-tuning under matched conditions.Source: §1 · p. 3
  • The benchmark makes conveyor velocity, bounded execution duration, and geometric completion requirements controllable for systematic evaluation individually and in combination.The conditions couple spatial completion requirements with finite execution windows.Source: §1 · p. 3
  • Performance degrades as industrial constraints are introduced and combined, while LIBERO-Industry co-training achieves higher observed target-transfer completion than original-LIBERO under a shared protocol.The evaluation covers five generalist policies, including VLA models and a WAM.Source: §1 · p. 3

2 Related Work

Prior work studies generalist robot policies, manipulation benchmarks, and manufacturing-oriented evaluation, but industrial operating conditions for generalist policies remain underexplored. LIBERO-Industry frames these conditions as controlled variables in conveyor-transfer tasks.

2.1 Generalist Robot Policies

  • Generalist policy work spans discrete-token VLA action decoding, continuous action chunks, and WAMs that predict future states with actions, but calibrated industrial behavior remains underexplored.Other approaches condition actions on predictive visual representations or use video prediction and future-observation embeddings.Source: §2.1 · p. 3

2.2 Generalist Robot Manipulation Benchmarks

  • Manipulation benchmarks address multi-task learning, long-horizon execution, transfer, and robustness, while LIBERO-Industry adds controllable production-line conditions to conveyor-transfer tasks.The benchmark uses finite execution windows, task-specific placement predicates, and repeated multi-object handling.Source: §2.2 · pp. 3–4, Table 1 · p. 4

2.3 Industrial Process-Aware Evaluation

  • Manufacturing benchmarks cover workflows, industrial manipulation, and logistics, but neither they nor embodied-manipulation evaluations systematically test generalist policies under controlled production-line conditions.LIBERO-Industry exposes domain-specific process conditions across standardized conveyor-transfer tasks.Source: §2.3 · pp. 4–5

3 LIBERO-Industry Benchmark Design

LIBERO-Industry evaluates material-transfer policies with explicit task predicates and transport-derived episode horizons, structuring industrial constraints across task generation, workflow suites, and evaluation metrics.

3 LIBERO-Industry Benchmark Design

  • Each task instance receives explicit completion predicates and an episode horizon derived from its transport schedule.Source: §3 · p. 5

3.1 Preliminaries and Task Interface

  • The language-conditioned interface supplies instructions and visual-proprioceptive observations, while evaluator-only simulator state supports task-predicate computation.Source: §3.1 · p. 5

3.2 Industrial Process Constraints

  • Transport states change source or destination geometry, while temporal opportunities bound when each workpiece transfer remains feasible.Source: §3.2 · pp. 5–6

3.3 Process-Aware Procedural Task Generation

  • The generator conditions object initialization and task timing on material flow, serializes layouts as BDDL, and resamples infeasible layouts.Source: §3.3 · pp. 6–7, Figure 2 · p. 6

3.4 Task Suites and Difficulty Levels

  • The suites cover moving-source and/or moving-destination transfers, with task completion requiring acquisition and placement within the relevant opportunities.Source: §3.4 · p. 7

3.5 Evaluation Protocol and Metrics

  • Evaluation reports binary task success and partial-completion rates, including a score that also penalizes displacement of pre-existing destination workpieces.Source: §3.5 · p. 8

4 Experiments

The experiments evaluate five policies on held-out benchmark tasks, diagnose performance under cumulative industrial stress, and test physical D2S co-training. Across these evaluations, complete-task success was limited, while partial-transfer and real-world comparisons exposed differences between policies and training conditions.

4.1 Experimental Setup

  • The deterministic evaluation covered 36 held-out tasks, with four tasks per difficulty level in each of S2D, D2S, and D2D, and 10 initial states per task.For physical D2S trials, S1 and S2 shared the same 95 physical demonstrations and training procedure, while their simulation-data source and corpus size differed.Source: §4.1 · p. 8, §4.4 · p. 9

4.2 Benchmark Results

  • Every checkpoint had 0.0% SR (0/360), while π0.5 alone recorded PCRT = 0.4% overall and the highest PCRS = 12.5%.The mean initial-state slotting baseline was PCRinit = 12.31% (12.3% at Table 3 precision); PCRS can reflect pre-existing valid destination workpieces, whereas PCRT isolates target transfers.Source: Table 3 · p. 9, §4.2 · p. 9

4.3 Ablation and Findings

  • For π0.5, SR fell from 97.0% at D0 to 0.0% at D6, with reach-stage failures localizing the bottleneck to longer multi-object sequences.The cumulative progression is diagnostic rather than causal; partial transfer persisted at D5–D6.Source: §4.3 · p. 9, Figure 3 · p. 10

4.4 Real-World Experiments

  • At 0.03 m/s, S2 achieved PCRT = 34.7% (25/72) versus S1’s 26.4% (19/72); at 0.05 m/s, results were 23.6% (17/72) versus 4.2% (3/72).The physical D2S evaluation used 72 planned placements per policy–speed condition.Source: §4.4 · p. 10
  • Across five physical operating-condition variations, S2 exceeded S1 by 12.5–34.7 percentage points, with the largest difference under camera variation (34.7% versus 0.0%).The tested variations included illumination, object appearance, camera configuration, and table texture.Source: §4.4 · p. 10

5 Conclusion and future work

LIBERO-Industry structures industrial process constraints into controlled S2D, D2S, and D2D task suites. Results show performance degradation under cumulative stress and higher observed D2S manipulation ability and generalization with LIBERO-Industry co-training; future evaluation should better reflect production conditions.

5 Conclusion and future work

  • LIBERO-Industry extends LIBERO with suites for product variability, material flows, finite execution windows, and tight geometric tolerances.The suites provide controlled variations in task configuration and operating conditions.Source: §5 · pp. 10–11
  • Results show performance degradation across cumulative industrial stress and higher D2S manipulation ability and generalization for LIBERO-Industry co-training than for original-LIBERO co-training.The benchmark organizes controlled task-configuration and operating-condition variations into S2D, D2S, and D2D suites.Source: §5 · pp. 10–11
  • Future evaluation should calibrate cycle times to empirical operating rates and test repeated material arrivals, recovery events, and long-horizon throughput.The benchmark’s finite opportunity windows abstract heterogeneous factory needs; industrial readiness should also be assessed for compatibility with diverse production processes.Source: §5 · p. 11

A Control Paradigms and Technical Comparison

The section compares classical control, task-specific learning, and generalist robot policies through their deployment dependencies and limitations. Its framing motivates high-fidelity simulation environments such as LIBERO-Industry.

A Control Paradigms and Technical Comparison

  • Classical and modular control can execute repeatably under calibrated conditions, but depend on accurate models, engineered task logic, and integrated perception, planning, and control.These stacks separate sequencing, motion planning, trajectory generation, and feedback control.Source: A Control Paradigms and Technical… · p. 18
  • Task-specific imitation and reinforcement learning reduce manual motion specification, but remain tied to the training task, embodiment, interface, reward design, and data coverage.The section names ACT, Diffusion Policy, and model-free reinforcement learning as examples.Source: A Control Paradigms and Technical… · p. 18
  • Generalist policies broaden task and semantic coverage through multi-task or multi-embodiment pretraining, while runtime frequency, contact behavior, and precision remain dependent on architecture and deployment choices.World Action Models additionally predict future visual dynamics alongside action sequences.Source: A Control Paradigms and Technical… · p. 18

B.1 Task Suites, Assets, and Fixtures

LIBERO-Industry organizes industrial transfer tasks around three transport scenarios, varied fixtures, and systematically specified task conditions. Its catalog, instructions, timing, and episode horizons define reproducible tasks with spatial goals and finite opportunities.

B.1 Task Suites, Assets, and Fixtures

  • The benchmark encodes each task as a BDDL specification with objects, initial conditions, a language instruction, and goal propositions grounded by benchmark predicates.The three transport suites are Static-to-Dynamic loading, Dynamic-to-Static unloading, and Dynamic-to-Dynamic transfer.Source: §B.1 · p. 18, Figure 5 · p. 19
  • Fixtures progress from domain-wide sector trays to object-specific grooves and form-fitting inlays, with a strict 1 cm geometric clearance rule.Grooves use workpiece bounding-box dimensions plus the margin; inlays use proportionally expanded negative-space profiles.Source: §B.1 · p. 20

B.2 Catalog Composition and Spatial Distribution

  • The catalog specifies targets, confounders, and placement regions by suite and level; its declared region centers are largely within the LIBERO-Spatial center range.The range contains 510 of 528 Industry region declarations.Source: §B.2 · p. 21, Figure 7 · p. 21

B.3 Language Instruction Construction and Perturbation

  • Task instructions are authored offline from structured task metadata, then perturbed by level-specific rules while preserving the action, target, and destination.Finalized instructions are stored as fixed mappings from BDDL task identifiers and retrieved deterministically at runtime.Source: §B.3 · p. 22, Table 6 · p. 22

B.4.1 Terminology and Scope

  • The benchmark reports arrival intervals, service-opportunity intervals, deadlines, and episode horizons rather than equating them with takt time or with one another.An episode horizon limits rollout duration, while arrival and service-opportunity intervals specify workload presented to the controller.Source: §B.4.1 · p. 22

B.4.2 Suite-Specific Physical Timing

  • Source arrival timing depends on conveyor velocity and workpiece pitch, while destination service timing depends on slot pitch, transfer velocity, and stationary dwell.D2D retains source and destination constraints as an ordered pair rather than collapsing asynchronous lines into one interval.Source: §B.4.2 · pp. 22–23

B.4.3 Suite-Specific Episode Horizons

  • Suite-specific horizon expansions account for transport events and task allowances, and each reports both continuous rollout duration and the corresponding integer control-step count.The documented fixed values are toff = 1.5 s, ti = 1.0 s, tr = 3.0 s, tx = 1.5 s, tb = 1.5 s, and fc = 20 Hz.Source: §B.4.3 · pp. 23–24

B.4.4 Evaluation Freeze and Throughput Reporting

Evaluation freezes task timing and success conditions across policies, while the simulator implements moving transport and level-dependent completion constraints. Reported opportunity rates are distinct from observed good-part throughput, and the standards provide reporting context rather than validating simulator bounds.

B.4.4 Evaluation Freeze and Throughput Reporting

  • The protocol freezes task, timing, deadline grid, horizon, control frequency, seeds, settling dwell, and success predicate before policy comparison, and reports success-deadline curves with feasibility flags.No timing value is retuned for an individual policy.Source: §B.4.4 · p. 24
  • Intervals of 4, 5, and 6 s correspond to 900, 720, and 600 theoretical opportunities/hour, respectively, but these are not good-part throughput.Observed good-part throughput is reported as a benchmark-conditioned measurement, not a universal HMLV production rate.Source: §B.4.4 · pp. 24–25

B.5.1 Simulation Mechanics and Transport Physics

  • Un grasped workpieces in contact with an active conveyor are synchronized to the belt transport vector, while actively grasped objects are excluded from that update.The simulation applies the conveyor velocity using the mechanism rotation, localized travel axis, and belt speed.Source: §B.5.1 · p. 25

B.5.2 Kinematic Overrides for Conveyor and Assembly Physics

  • Kinematic overrides address simulation chattering and micro-slippage, anchoring workpieces to conveyors and applying staged slot locks for terminal placement.The described lock levels range from planar locking at Level 1 to full 3D spatial locking at Levels 2 and 3.Source: §B.5.2 · pp. 25–26

B.5.3 Predicate Scaling and Constraint Matrix

  • As environmental variance increases across difficulty levels, task predicates impose stricter kinematic constraints and require higher degrees of freedom for successful execution.The constraint matrix marks which mechanisms are strictly verified for task success and which are relaxed or not applicable.Source: §B.5.3 · p. 27, Table 10 · p. 27, Table 11 · p. 27

B.6.1 Standards and Reporting Basis

  • ISO 9283, ISO 8995-1, and GB/T 10595 provide terminology and reporting context, but do not prescribe or validate the simulator-domain perturbation bounds.Those standards respectively frame pose accuracy and repeatability, workplace lighting, and commanded belt-conveyor conditions.Source: §B.6.1 · p. 27

B.6.2 Perturbation Specifications

LIBERO-Industry defines three generalization tiers through perturbations to workspace geometry, visual conditions, and transport dynamics. Its seeded task records support reproducibility, while the checkpoint evaluation measures cross-benchmark transfer from original-LIBERO fine-tuning.

B.6.2 Perturbation Specifications

  • Level 1 is nominal, while Levels 2 and 3 provide moderate and broad generalization conditions through changes to geometry, observations, and transport.The specified variations include placement and fixture poses, camera and illumination settings, conveyor speeds, and service times.Source: §B.6.2 · pp. 27–28

B.6.3 Reproducibility and Configuration

  • Scene perturbations are deterministic for each task seed, and the generator serializes camera, lighting, surface, reset, and rollout parameters.The configuration also records suite-specific geometry, composition, and transport rules.Source: §B.6.3 · p. 28

C.1 Evaluated Models and Checkpoints

  • The five evaluated policies are publicly released checkpoints fine-tuned on original LIBERO without further LIBERO-Industry fine-tuning, so this is cross-benchmark transfer rather than base-model evaluation.The paper says exact weight revisions and model-specific inference settings should be recorded in evaluation configurations.Source: §C.1 · p. 29, Table 13 · p. 29

C.2 Evaluation-Set Breakdown

The appendices document the held-out evaluation and cumulative stress diagnostics, then describe cross-embodiment conversion and recovery and the physical D2S data collection.

C.2 Evaluation-Set Breakdown

  • Table 14 describes the 36-task held-out split, and Table 15 reports suite-level scores for five checkpoints over 120 rollouts per policy per suite.Table 14 also reports task-composition counts and a macro-average of task-level ratios.Source: Table 15 · p. 29, Table 14 · p. 29, §C.2 · p. 29

C.3.2 Trajectory-Level Failure Classification

  • The classifier labels each rollout by its first unmet phase in the reach–grasp–lift–transport–place sequence, with global or fallback outcomes for other cases.Representative failed-rollout frames illustrate the categories but are not exhaustive.Source: §C.3.2 · p. 30, Figure 8 · p. 30, §C.3.3 · p. 31

C.3.1 Stress-Progression Setup

  • The D0–D6 progression cumulatively adds industrial backgrounds, removes a visual anchor, changes trays, adds conveyor dynamics, and expands tasks from one to three objects.Adjacent stages may change multiple factors, so the progression localizes breakdowns without isolating causes; numeric evaluator seeds were not retained for prespecified confidence-interval analysis.Source: §C.3.2 · p. 31

D Cross-Embodiment Conversion and Policy Adaptation

  • Panda-to-AUBO conversion tracks Cartesian references rather than joint coordinates, aligning task-space motion without assuming pixelwise camera equivalence.The Panda and AUBO arms have seven and six joints, respectively; corresponding task-body poses are required to agree within 10^-6 m and 10^-3 degrees.Source: D Cross-Embodiment Conversion and… · p. 31, Figure 10 · p. 31

D.1 Trajectory Conversion and Recovery

  • The recovery procedure transforms object-centric segments into AUBO reference frames and retains candidates only after task, action, controller, and three-replay checks pass.Accepted recoveries retain demonstration provenance and are merged only with replay-verified raw conversions.Source: §D.1 · p. 33, Figure 11 · p. 33, Table 17 · p. 34

E.1 Physical Workcell and Sensor Setup

  • The physical D2S workcell transfers objects from a moving conveyor to stationary fixtures, with tray, groove, or inlay targets across three levels.The accepted physical dataset contains 95 demonstrations and 41,257 frames; paired external and wrist-camera views are recorded, and one episode was excluded for failing the 25 ms synchronization threshold.Source: §E.1 · p. 34, Figure 12 · p. 34, Figure 13 · p. 36

E.2 Real-World Collection, Splits, and Quality Control

  • The physical collection varies object combinations and target-slot permutations, reserving two disjoint permutations per combination for held-out evaluation.Conveyor speed is balanced rather than fully crossed with every factor, and a task configuration is distinct from a recorded trajectory.Source: §E.2 · p. 35, Table 18 · p. 35

E.4 Sim–Real Co-Training Configuration

The co-training configuration compares original-LIBERO and LIBERO-Industry simulation data while sharing physical demonstrations and configured training settings.

E.4 Sim–Real Co-Training Configuration

  • S1 combines original-LIBERO recovered-v2 simulation demonstrations with physical demonstrations, while S2 substitutes LIBERO-Industry simulation demonstrations and keeps the same physical dataset.Both runs use the same 95 physical demonstrations.Source: §E.4 · p. 37, Table 19 · p. 37
  • Both runs sample LeRobot frames at a 9:1 simulation-to-real ratio and are configured for 30,000 optimizer updates with global batch size 252.The ratio specifies a sampling distribution, not the proportion of unique trajectories.Source: §E.4 · p. 38, Table 20 · p. 38
  • The comparison matches physical-data source, sampling weights, and configured optimization budget, but not the sizes of the unique simulation datasets.Source: §E.4 · p. 38

Related papers

Loading 2610.16782v1…