PaperlayerView exact revision on arXiv

Source-linked AI summary

RB2: Robotic Manipulation Benchmarking with a Twist

Sudeep Dasari, Jianren Wang, Joyce Hong, Shikhar Bahl, Yixin Lin, Austin Wang, Abitha Thankaraj, Karanbir Chahal, Berk Calli, Saurabh Gupta, David Held, Lerrel Pinto, Deepak Pathak, Vikash Kumar, Abhinav Gupta

arXiv:2203.08098v2cs.RO

TL;DR

Robotic manipulation lacks a benchmark that is both broadly accessible and reproducible across variable physical setups. RB2 pairs tasks and protocols with state-of-the-art baseline implementations, then pools local rankings across labs. The evaluations show that simple open-loop behavior cloning can outperform more complex methods, while raw performance numbers do not generalize across labs.

  • Problem

    Robotic manipulation benchmarks must balance broad accessibility with reproducibility because physical setups vary across labs.

    Source: §1 · p. 2

  • Method

    RB2 pairs manipulation tasks and experimental protocols with state-of-the-art baselines, using local rankings and pooled multi-lab evaluations to form global rankings.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    BC-Open frequently outperforms more complicated methods, while raw performance numbers do not generalize across labs.

    Source: §6.1 · p. 9, §6.2 · p. 10

  • Takeaways & Limitations

    RB2 provides global rankings and implementations that users can consult to select baselines and demonstrate statistically significant local improvements.

    Source: §7 · p. 10

    The provided vision representation pipeline is a workaround for manipulation researchers; learning general 3D object representations from weak supervision remains unsolved.

    Source: §5.4 · p. 7

Abstract

from arXiv · show

Benchmarks offer a scientific way to compare algorithms using objective performance metrics. Good benchmarks have two features: (a) they should be widely useful for many research groups; (b) and they should produce reproducible findings. In robotic manipulation research, there is a trade-off between reproducibility and broad accessibility. If the benchmark is kept restrictive (fixed hardware, objects), the numbers are reproducible but the setup becomes less general. On the other hand, a benchmark could be a loose set of protocols (e.g. object sets) but the underlying variation in setups make the results non-reproducible. In this paper, we re-imagine benchmarking for robotic manipulation as state-of-the-art algorithmic implementations, alongside the usual set of tasks and experimental protocols. The added baseline implementations will provide a way to easily recreate SOTA numbers in a new local robotic setup, thus providing credible relative rankings between existing approaches and new work. However, these local rankings could vary between different setups. To resolve this issue, we build a mechanism for pooling experimental data between labs, and thus we establish a single global ranking for existing (and proposed) SOTA algorithms. Our benchmark, called Ranking-Based Robotics Benchmark (RB2), is evaluated on tasks that are inspired from clinically validated Southampton Hand Assessment Procedures. Our benchmark was run across two different labs and reveals several surprising findings. For example, extremely simple baselines like open-loop behavior cloning, outperform more complicated models (e.g. closed loop, RNN, Offline-RL, etc.) that are preferred by the field. We hope our fellow researchers will use RB2 to improve their research's quality and rigor.

1 Introduction

Robotic manipulation benchmarks face a trade-off: tightly controlled setups can limit generality, while loosely specified ones undermine reproducibility. RB2 pairs manipulation tasks and protocols with state-of-the-art baselines, supporting local comparisons and evaluations across labs.

1 Introduction

  • RB2 provides experimental, training, and evaluation procedures alongside implementations of five offline-track baseline algorithms.The benchmark focuses on ranking general-purpose manipulation algorithms across tasks rather than maximizing performance on individual tasks.Source: Figure 1 · p. 2, §1 · p. 3
  • RB2 addresses the tension between broad accessibility and reproducibility by comparing methods within local setups and pooling local rankings across labs.The same pouring experiment produced performance metrics that differed by 20% between two lab spaces.Source: §1 · p. 2
  • Its four SHAP-inspired daily-living tasks are pouring, scooping, zipping, and insertion, documented with train and test objects and objective performance metrics.The tasks are designed for standard two-finger grippers and commodity robot hardware.Source: §1 · p. 2, Figure 1 · p. 2
  • The authors evaluate the same set of algorithms across two labs to demonstrate the credibility of the benchmark framework.The passage does not report the resulting rankings or their values.Source: §1 · p. 3

2 Related Work

Prior real-world robotics benchmarks often control variation through tightly specified environments or hardware, which can restrict their audience. Simulation benchmarks are more common, but their simplified visuals and dynamics limit how well they represent real-world manipulation.

Robotics Benchmarks

  • Controlled real-world robotics benchmarks can be too rigid in their hardware and experimental requirements, limiting access for researchers without the specified equipment or interested in broader tasks.Low-cost platforms seek to democratize access but can be limited to simple manipulation tasks by their hardware and control capabilities.Source: §2 · p. 3
  • Standardized object sets such as YCB [9] control variation in object selection, but objects are only one part of a robotic experimental procedure.Other sources of variation include hardware, lighting, and the evaluation tasks themselves.Source: §2 · p. 3

Simulation Benchmarks

  • Simulation benchmarks are common, but simplified visuals and physics can miss real-world dynamics, complex object interactions, and hand-object contacts.These simplifications can let algorithms game tasks, making their effectiveness harder to infer in real-world settings.Source: §2 · p. 3

3 Benchmarking with a Twist

Robotic benchmarks cannot ship as identical evaluations across labs because real-world environment parameters vary. RB2 therefore combines local evaluations against shared baselines with rankings aggregated from multiple labs.

3 Benchmarking with a Twist

  • A conventional benchmark fixes task parameters and evaluation procedures so each method receives a reproducible score, but real-world robotics parameters cannot be tightly controlled across labs.Hardware, lighting, workspace size, and friction can vary between sites.Source: §3 · pp. 3–4
  • The ice-cream example compares a proposed method with a public SOTA baseline under the same warm-condition evaluation, rather than comparing scores from different conditions.The example distinguishes baseline conditions θc from warm conditions θw.Source: §3 · p. 4
  • RB2 defines a benchmark as tasks and prior SOTA baselines, then has researchers evaluate their method alongside those baselines under local environment parameters.Raw results may differ across labs, so the benchmark uses a fixed ranking function to compare methods locally.Source: §3 · p. 4
  • RB2 proposes combining local rankings from several labs into global rankings, with users uploading results to a central repository rather than each running experiments at multiple sites.The authors expect a truly better method to outperform baselines across the majority of environments.Source: §3 · p. 4

4 Benchmark Details - Tasks and Experimental Protocols

RB2 defines a shared observation-action recipe while allowing local environment differences, and combines local comparisons into global rankings. Its benchmark includes tasks, evaluation functions, and prior state-of-the-art policies.

Setup

  • Testing environments provide observations and accept actions applied for a fixed period, forming trajectories with a maximum horizon.The recipe requires all robot testing environments to follow this interface.Source: §4 · p. 4

Tasks (τ)

  • A task is a broad behavior category, while an instance is a sample that must have objective measures of trajectory success.For example, pouring is a task category, while scooping almonds from a particular box is an instance.Source: §4 · p. 4

Task/Environment Parameters (θ)

  • Workspace parameters, such as initial and goal object positions, are tightly controlled, while system parameters can vary subject to embodiment and workspace constraints.The stated constraints include one morphological embodiment, no sensor or actuation augmentation within the workspace, and a robot starting outside the task area.Source: §4 · pp. 4–5

Baselines and local rankings

  • RB2 treats state-of-the-art policies as integral benchmark baselines, with each policy producing actions from observations for task instances.Local rankings order methods by performance, with Tukey’s HSD pairwise testing used to test statistical significance and reduce ranking noise.Source: §4 · p. 5

Global Ranking

  • Plackett-Luce aggregates full or partial local rankings into global rankings by estimating method worths.The example workflow requires independent-lab verification before a proposed method joins the global ranking.Source: §4 · p. 5

5 Benchmark Instantiation: RB2

RB2 collects demonstrations, trains learned policies, and evaluates them across four household manipulation tasks using held-out objects and positions. In the reported comparison, open-loop behavior cloning often outperforms NDP and closed-loop baselines, while Offline RL is omitted because it never succeeds on any task.

5 Benchmark Instantiation: RB2; 5.1 Collection and Evaluation Protocols (m):

  • RB2 repeats three stages per task: collecting human demonstrations, fitting a baseline policy, and evaluating its control performance in test scenarios.Training instances are stratified over train objects and positions, and tests use held-out positions, test objects, and training objects with task-specific, consistently computed metrics.Source: §5 · p. 6, §5.1 · p. 6

5.2 Tasks and Workspace Parameters (τ, θw)

  • The benchmark applies its protocol to pouring, scooping, insertion, and zipping, household tasks loosely inspired by SHAP and spanning varied manipulation behaviors.The task measures include poured material, scooped material, zipped distance, and binary insertion success.Source: §5.2 · pp. 6–7

5.3 Baselines

  • RB2 uses a 30 Hz joint-position control stack with RGB and proprioceptive observations, and will release Franka Panda parameters and stacks for local prototyping.The released baseline system stacks are intended to support rapid prototyping in other researchers’ environments.Source: §5.3 · p. 7

5.4 Network Architecture and Representation Learning

  • A vision stack trains representations to predict object poses from images using 1,200 image-pose pairs per environment, then initializes learned policies.The authors provide the pipeline as a way for manipulation- and control-focused researchers to sidestep the unsolved challenge of learning general 3D object representations from weak supervision.Source: §5.4 · p. 7

5.5 Open-Loop Imitation

  • Open-loop behavior cloning predicts a horizon of expert actions from the initial observation and executes that trajectory without replanning.RB2 also implements NDPs, which embed dynamical-system structure to produce smoother trajectories.Source: §5.5 · p. 7
  • Open-Loop BC often outperforms NDP and closed-loop baselines across the four tasks, while Offline RL is omitted because it never succeeds on any task.Figure 3 reports metrics normalized to [0,1], with 1 indicating best possible performance.Source: Figure 3 · p. 8

5.6 Closed-Loop Imitation

  • Closed-loop behavior cloning predicts a new action from each current observation, addressing the open-loop failure to correct earlier mistakes.The recurrent baseline pairs closed-loop cloning with an LSTM trained by teacher forcing on trajectory snippets.Source: §5.6 · p. 8

5.7 Offline Reinforcement Learning

  • The Offline RL baseline implements MOReL, learning dynamics from static data and using an ensemble-based unknown-state detector to penalize uncertain regions.Its dataset combines optimal trajectories, noise-perturbed trajectories, and random interactions; planning uses MPC initialized with a behavior-cloned policy.Source: §5.7 · p. 8

6 Experiments

The experiments evaluate five baselines across four tasks and two distinct labs, examining task-level performance and local and pooled rankings. BC-Open consistently ranks above the other methods across settings, while raw performance numbers do not generalize across labs.

6.2 Method Rankings

  • BC-Open consistently ranks higher than the other methods across settings, emerging as true SOTA on these tasks.The rankings compile results across four tasks and local and global evaluations.Source: Table 1 · p. 9, §6.2 · p. 10

6.1 Metric Level Analysis

  • The full baseline suite was evaluated on all four tasks in two distinct Franka lab spaces, with normalized metrics and train- and test-object scores reported separately.The labs used the same robot hardware but different spaces, camera setups, and platforms; test-object performance was consistently lower than training-object performance.Source: §6.1 · p. 9
  • MOReL significantly under-performs on every task; adding noise-augmented demonstrations and random trajectories to pouring data did not boost its performance.The authors suggest offline RL may require more diverse negative data and conclude that it needs significant amounts of data to become competitive on RB2.Source: §6.1 · p. 9
  • Zipping is the easiest task, insertion the hardest, and pouring and scooping are medium difficulty; BC-Open frequently outperforms more complicated methods.BC-Closed and MOReL are among the more complicated methods that BC-Open frequently outperforms.Source: §6.1 · p. 9

6.2 Method Rankings

  • Raw performance numbers do not generalize across labs, so RB2 computes per-lab rankings and combines evaluations into global rankings with significance tests.The authors expect small deviations between local rankings because of hardware-experiment noise and the number of test objects.Source: §6.2 · p. 10
  • Local rankings can differ in significance across labs: pouring results favored NDP over BC-Closed and BC-LSTM significantly in Lab 1, but not significantly in Lab 2.RB2 proposes pooling evaluations to handle noise and support emergence of true SOTA.Source: §6.2 · p. 10
  • RB2’s central proposition is to pool lab data so the field can reach consensus despite noise that small academic labs cannot resolve alone.The authors describe this crowdsourced approach as democratizing access to robotics knowledge and helping SOTA emerge.Source: §6.2 · p. 10

7 Conclusion

The conclusion presents RB2 as a four-task benchmark with five baseline approaches and a mechanism for combining evaluations across labs. It reports that simple behavior cloning can outperform more complicated methods, including Offline RL.

7 Conclusion

  • RB2 pairs four manipulation tasks with five SOTA baselines and supports a singular global ranking by combining evaluation data from multiple labs.The tasks are pouring, scooping, zipping, and insertion; the baselines are BC-Open, NDP, BC-Closed, BC-LSTM, and MOReL.Source: §7 · p. 10
  • The authors report that closed-loop learning approaches often struggle in the real world and simple behavior cloning can outperform more complicated methods such as Offline RL.They say RB2 will evolve as local advancements are contributed, and plan further replications in more lab spaces and with other robots.Source: §7 · p. 10
  • The authors say RB2’s global rankings and implementations can help users choose baselines and demonstrate statistically significant improvements in their local labs.They hope the benchmark will democratize robotics and scale research progress.Source: §7 · p. 10

Related papers

Loading 2203.08098v2…