PaperlayerView exact revision on arXiv

Source-linked AI summary

FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learning

Jianlan Luo, Charles Xu, Fangchen Liu, Liam Tan, Zipeng Lin, Jeffrey Wu, Pieter Abbeel, Sergey Levine

arXiv:2401.08553v3cs.RO

TL;DR

Robotic learning still faces the challenge of combining generalization with physically intricate manipulation while keeping tasks accessible. FMB provides a real-world benchmark, demonstrations, and reusable policies for multi-stage functional manipulation. It spans single-object and multi-object assemblies, with 66 printable objects and 22,500 demonstrations.

  • Problem

    Robotic learning needs accessible, measurable tasks that combine generalization across varied objects with challenging contact dynamics and manipulation skills.

    Source: §1 · pp. 1–2

  • Method

    FMB builds real-world assembly tasks around functionally relevant sequences of grasping, repositioning, and insertion, with imitation policies for individual stages and full tasks.

    Source: §1 · pp. 2–3

  • Results

    FMB spans single-object and multi-object multi-stage assemblies and provides 66 replicable objects, 22,500 demonstrations, and pretrained policies.

    Source: §1 · p. 3, §6 · p. 15

  • Takeaways & Limitations

    FMB is a modular toolkit for studying individual manipulation skills, their sequencing, and full multi-stage tasks.

    Source: §1 · p. 3, §6 · p. 15

    The current assembly tasks include fewer boards than the object grammar could support.

    Source: §6 · p. 15

Abstract

from arXiv · show

In this paper, we propose a real-world benchmark for studying robotic learning in the context of functional manipulation: a robot needs to accomplish complex long-horizon behaviors by composing individual manipulation skills in functionally relevant ways. The core design principles of our Functional Manipulation Benchmark (FMB) emphasize a harmonious balance between complexity and accessibility. Tasks are deliberately scoped to be narrow, ensuring that models and datasets of manageable scale can be utilized effectively to track progress. Simultaneously, they are diverse enough to pose a significant generalization challenge. Furthermore, the benchmark is designed to be easily replicable, encompassing all essential hardware and software components. To achieve this goal, FMB consists of a variety of 3D-printed objects designed for easy and accurate replication by other researchers. The objects are procedurally generated, providing a principled framework to study generalization in a controlled fashion. We focus on fundamental manipulation skills, including grasping, repositioning, and a range of assembly behaviors. The FMB can be used to evaluate methods for acquiring individual skills, as well as methods for combining and ordering such skills to solve complex, multi-stage manipulation tasks. We also offer an imitation learning framework that includes a suite of policies trained to solve the proposed tasks. This enables researchers to utilize our tasks as a versatile toolkit for examining various parts of the pipeline. For example, researchers could propose a better design for a grasping controller and evaluate it in combination with our baseline reorientation and assembly policies as part of a pipeline for solving multi-stage tasks. Our dataset, object CAD files, code, and evaluation videos can be found on our project website: https://functional-manipulation-benchmark.github.io

1. Introduction

Robotic learning needs accessible tasks that combine object generalization with physically challenging manipulation. FMB addresses this through real-world, multi-stage assembly tasks and reusable data, objects, and policies.

1. Introduction

  • The benchmark targets the gap between broad generalization on simple skills and physically complex skills studied without extensive generalization.The authors frame the challenge as combining both while keeping tasks constrained enough for meaningful progress.Source: §1 · pp. 1–2
  • FMB defines functional manipulation as arranging behaviors such as grasping and repositioning so objects can be used in subsequent physical interactions.Its tasks range from single-object pick-and-place to long-horizon, multi-stage assemblies of multiple parts.Source: §1 · p. 2
  • Tasks vary object shape, size, color, and location; solving them requires suitable grasps, reorientation, compliant insertion, skill sequencing, and recovery from failures.Multi-object assemblies add interlocking parts that must be manipulated in the right order.Source: Figure 1 · p. 2, §1 · p. 3
  • FMB provides 66 replicable 3D-printed objects, 22,500 human demonstrations, multimodal recordings, and pretrained imitation policies for individual stages or full tasks.The recorded modalities include RGB and depth images from multiple cameras, robot kinematics, and end-effector force/torque measurements.Source: §1 · p. 3

2. Related Work

Prior work often studies generalization on simpler manipulation or physical complexity on narrower tasks, while datasets and object sets offer limited support for accessible, multi-stage generalization studies. FMB targets this combination with tasks spanning grasping, repositioning, and assembly.

2. Related Work

  • Generalization is often studied with simpler tasks, while physically challenging manipulation is often studied without extensive generalization.The authors describe fewer works as combining these factors and position FMB as a scoped benchmark for their intersection.Source: §2 · p. 3
  • Accessible standard object sets have rarely been paired with multi-stage generalization tasks, and prior 3D-printed benchmarks typically focus on grasping rather than all manipulation stages.FMB's parts are intended for evaluating grasping, reorientation, and assembly.Source: §2 · pp. 3–4
  • Existing demonstration datasets often emphasize basic pick-and-place, while insertion datasets focus on narrower skills; FMB covers grasping, repositioning, and generalization.The authors distinguish FMB from work targeting best-in-setting performance on narrow tasks such as ultra-high-precision industrial insertion.Source: §2 · pp. 3–4

3. Functional Manipulation Benchmark

FMB organizes single-object and multi-object assembly tasks around individual skills and their composition, with evaluation focused on generalization across object locations and physical attributes. It provides printable objects, demonstrations, and protocols for studying those skills and full assembly tasks.

3. Functional Manipulation Benchmark

  • FMB covers single-object and multi-object multi-stage assembly, evaluating individual-skill generalization and performance on full tasks.The two categories share design principles, but multi-object tasks add the need to select the appropriate object for manipulation.Source: §3 · p. 4

3.1. Object Set

  • The object set contains 66 printable objects, including procedurally generated interlocking assemblies and five test objects varying from training objects in shape, size, and color.The 54 single-object objects span nine shapes, six sizes per shape, and eight colors; mating tolerances are 1–2 mm.Source: §3.1 · p. 4

3.2. Individual Manipulation Skills

  • Functional grasping chooses poses that facilitate downstream manipulation; repositioning may use a fixture and regrasping for reorientation; insertion requires fine-grained precise manipulation to place diverse-shaped objects into matching slots.The dataset includes 4,500 placing-and-regrasping demonstrations and 6,750 single-object assembly demonstrations from various initial conditions.Source: §3.2 · pp. 4–5

3.3. Single-Object Multi-Stage Manipulation Tasks

  • Single-object multi-stage tasks compose grasping, reorientation, and insertion, where errors can compound when an object's resulting pose differs from insertion demonstrations.The object-set figures include unseen test objects for evaluating generalization to new shape, size, and color combinations.Source: §3.3 · pp. 5–6, Figure 6 · p. 6

3.4. Multi-Object Multi-Stage Manipulation Tasks

  • Multi-object assemblies add target selection among nearby pieces, contact with already inserted objects, and the need to insert interlocking pieces in a specific order.The illustrated initial insert distributions vary the number of objects already on the board.Source: §3.4 · p. 6, Figure 7 · p. 6

3.5. Robotic System and Data Collection

  • The benchmark pairs a Franka-based robotic setup with 22,550 demonstration trajectories, each approximately five seconds long.Four cameras record RGB and depth, and the system logs end-effector force/torque information using the robot's sensing capabilities.Source: §3.5 · pp. 6–7, Table 1 · p. 7

3.6. Using the Benchmark

  • Users reproduce the setup and print the objects and tools, but can use FMB components to collect their own data with a different robot.The benchmark also releases the operating software and low-level controller.Source: §3.5 · p. 6, §3.6 · p. 8

3.7. Evaluation protocol

  • Evaluation protocols specify initial poses to test generalization consistently; single-object tests include held-out or novel objects, while multi-object tests vary starting arrangements and fix the board.Single-object protocols also specify starting poses for the object and insertion board.Source: §3.7 · p. 8

4. An Imitation Learning System for the FMB

The paper presents imitation-learning policies as baselines and reusable components for FMB tasks. It describes ResNet and Transformer policies for individual skills and uses a human-provided sequence to compose primitives for long-horizon tasks.

4. An Imitation Learning System for the FMB

  • The imitation-learning system provides baseline performance and components that researchers can extend for FMB tasks.The authors train and test ResNet and Transformer models on individual skills using the FMB dataset and evaluation protocol.Source: §4 · p. 8, §4.1 · p. 8

4.1. Imitation Learning Policies for Individual Skills

  • The ResNet policy encodes multiple RGB and depth views with weight-shared ResNets, combines them with proprioception and optional object and skill IDs, and predicts end-effector actions.Its MLP policy head represents a Gaussian distribution; the output includes a 6D end-effector twist and a gripper open-or-close variable.Source: §4.1 · pp. 8–9, Figure 9 · p. 9
  • The Transformer policy uses weight-shared image encoders, FiLM conditioning on object or primitive IDs, and self-attention over image and proprioception tokens to predict discretized actions.Each continuous action dimension is quantized into 256 bins during training and converted back to continuous values at runtime.Source: §4.1 · p. 9, Figure 9 · p. 9

4.2. Composing Skills to Solve Long-Horizon Tasks

  • For long-horizon tasks, a human-provided sequence triggers low-level primitives and can support recovery, while learning a high-level policy to choose primitives remains future work.Repeating grasp attempts or repositioning after an unsuccessful grasp are examples of recovery behaviors.Source: §4.2 · pp. 9–10

5. Experiments

The experiments compare imitation-learning architectures and input choices across grasping, repositioning, insertion, and multi-stage tasks, assessing generalization and hierarchical versus flat policies. They establish baselines and examine task difficulties and failure modes.

5.1. Grasping Task

  • 28/50 for Transformer and 27/50 for ResNet with full RGB data; ResNet fell to 12/50 at 20% data, while depth improved same-data RGB baselines.ResNet grasped unseen objects with success rates comparable to seen objects across data amounts.Source: §5.1 · pp. 10–11, Figure 10 · p. 10

5.2. Repositioning Task

  • ResNet without action chunking outperformed the other tested policies on rotate, while Transformer policies outperformed both ResNet variants on place on fixture and regrasp.Evaluations covered seen and unseen objects; the reported failure modes included mistimed gripper actions and rotating in the wrong direction.Source: §5.2 · p. 11, Figure 11 · p. 11

5.3. Insertion Task

  • Force/torque information was crucial for insertion, and the best input combined RGB views, TCP pose and velocity, and force/torque; adding depth deteriorated performance.The authors suggest noisy depth may confuse accurate end-effector pose readings when TCP pose is already available.Source: §5.3 · p. 11, Table 2 · p. 11
  • Insertion success decreased as object shape became more complex, with the three-prong object hardest and shape-dependent difficulty providing a benchmark gradient.For asymmetrical two-pronged objects, the required rotation can range from 0 to 90 degrees depending on grasp pose.Source: §5.3 · pp. 11–12, Figure 12 · p. 12
  • Across nine insertion shapes, unconditional ResNet achieved 7/45, shape-conditioned ResNet 14/45, and shape-conditioned Transformer 27/45 successes.The unconditional policy sometimes selected the wrong slot; the authors hypothesize Transformer attention and FiLM conditioning helped it attend to shape information.Source: §5.3 · p. 12, Table 3 · p. 12

5.4. Multi-Stage Manipulation Tasks

  • 19/30 single-object and 7/10 multi-object successes were the hierarchical policies’ maxima, while flat policies, including diffusion, scored 0/10 on both tasks.These outcomes support the authors’ view that FMB’s multi-stage tasks require addressing error compounding; they do not rule out multimodal demonstrations as a contributing factor.Source: Table 4 · p. 13, §5.4 · pp. 13–14, Table 5 · p. 14
  • For single-object tasks, Transformer hierarchies scored 19/30 versus 15/30; chaining separate ResNet policies scored 18/30 versus 0/30 for conditioned ResNet.For multi-object tasks, conditioned Transformer reached 7/10 versus 5/10 for conditioned ResNet.Source: §5.4 · p. 14
  • ResNets degraded when task-irrelevant camera inputs produced incorrect feature associations, whereas Transformers learned to ignore irrelevant modalities.A distant camera struggled to locate holes on a randomized assembly board; the additional view did not impair performance when the board position was fixed.Source: §5.4 · p. 14

6. Discussion and Limitations

FMB is presented as a reproducible benchmark for studying manipulation complexity and object generalization, with tasks spanning single-object skills through multi-object assemblies. Its dataset and open system support researchers’ method development, and the task design can be extended.

6. Discussion and Limitations

  • FMB combines task complexity, object generalization, and reproducibility to study critical challenges in robotic manipulation learning.The benchmark includes a comprehensive dataset and reproducible hardware and software system.Source: §6 · p. 15
  • Tasks range from single-object manipulation to multi-object, multi-stage assembly, using diverse 3D-printed objects that support generalization to unseen objects and task replication.The open-sourced imitation-learning system and experimental analysis offer a foundation for developing and evaluating methods.Source: §6 · p. 15
  • FMB’s assembly-object grammar could support more assembly-board varieties, while researchers can add objects and demonstrations to enrich the benchmark.The authors invite researchers to replicate the setup and try pretrained models as a starting point for developing and evaluating methods.Source: §6 · p. 15

Related papers

Loading 2401.08553v3…