PaperlayerView exact revision on arXiv

Source-linked AI summary

FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation

Minho Heo, Youngwoon Lee, Doohyun Lee, Joseph J. Lim

arXiv:2305.12821v1cs.ROcs.AIcs.LG

TL;DR

Current real-world manipulation benchmarks mainly test simple behaviors, leaving complex long-horizon autonomy insufficiently evaluated. FurnitureBench addresses this gap with a reproducible furniture-assembly benchmark, supporting resources, and simulation; evaluations show common IL and offline RL methods struggle with its long horizon and complex skills.

  • Problem

    Real-world manipulation benchmarks have largely limited RL, IL, and TAMP evaluation to simple behaviors, while complex long-horizon tasks require broader reasoning and dexterous skills.

  • Method

    FurnitureBench provides a reproducible real-world furniture-assembly benchmark with 3D-printable models, setup tools, demonstrations, systematic initialization, skill evaluations, and FurnitureSim.

  • Results

    Common imitation learning and offline reinforcement learning methods struggle to solve furniture assembly; in full assembly, they achieve up to 2 out of 12 subtasks on average.

  • Takeaways & Limitations

    FurnitureBench offers a standardized setting for identifying and comparing challenges in complex long-horizon real-world manipulation.

  • Takeaways & Limitations

    The benchmark uses a single Franka Emika Panda arm, limiting dexterity and interactions; extending it to multi-arm or multi-robot collaboration remains future work.

Abstract

from arXiv · show

Reinforcement learning (RL), imitation learning (IL), and task and motion planning (TAMP) have demonstrated impressive performance across various robotic manipulation tasks. However, these approaches have been limited to learning simple behaviors in current real-world manipulation benchmarks, such as pushing or pick-and-place. To enable more complex, long-horizon behaviors of an autonomous robot, we propose to focus on real-world furniture assembly, a complex, long-horizon robot manipulation task that requires addressing many current robotic manipulation challenges to solve. We present FurnitureBench, a reproducible real-world furniture assembly benchmark aimed at providing a low barrier for entry and being easily reproducible, so that researchers across the world can reliably test their algorithms and compare them against prior work. For ease of use, we provide 200+ hours of pre-collected data (5000+ demonstrations), 3D printable furniture models, a robotic environment setup guide, and systematic task initialization. Furthermore, we provide FurnitureSim, a fast and realistic simulator of FurnitureBench. We benchmark the performance of offline RL and IL algorithms on our assembly tasks and demonstrate the need to improve such algorithms to be able to solve our tasks in the real world, providing ample opportunities for future research.

I. INTRODUCTION

FurnitureBench addresses the lack of reproducible benchmarks for complex, long-horizon real-world manipulation by using furniture assembly as a standardized task suite. It provides reproducible hardware, furniture models, demonstrations, simulation, and evaluations showing that current learning methods still struggle with the task.

  • Existing reproducible manipulation benchmarks largely remain limited to simple, short-horizon tasks such as pick-and-place.
  • Furniture assembly combines hierarchical long-horizon reasoning with path planning and dexterous skills including grasping, alignment, insertion, and screwing.
  • FurnitureBench provides a reproducible real-world benchmark for evaluating RL, IL, and TAMP on complex assembly beyond pick-and-place.
  • The benchmark lowers setup barriers through 3D-printable furniture, comprehensive environment and evaluation guides, and 200+ hours of teleoperation demonstrations.
  • Single-skill evaluations identify insertion and screwing as challenging, while full-assembly methods achieve up to 2 out of 12 subtasks on average.
  • FurnitureSim supplies a simulated version of the benchmark to support fast iteration of experiments with new methods.

II. RELATED WORK

Existing robotic manipulation benchmarks span simulated and real-world settings, but reproducible real-world evaluations remain focused largely on simple, short-horizon tasks. FurnitureBench is positioned against this gap by combining complex assembly evaluation with standardized benchmarking resources.

  • Simulated manipulation benchmarks commonly evaluate picking, placing, in-hand manipulation, door opening, peg insertion, or screwing.
  • Composite simulated benchmarks add longer-horizon tasks, but often limit variation in object shapes and physical properties or abstract away low-level control.
  • FurnitureBench combines reproducibility, complex long-horizon evaluation, a large dataset, and a simulator for benchmarking.
  • Real-world manipulation benchmarks address challenges such as collisions, noisy and delayed sensing, environment resetting, and reward provision.
  • Many reproducible real-world benchmarks still focus on simpler short-horizon tasks such as stacking blocks, while NIST Assembly Task Boards lack a standard experimental setup.

III. FURNITURE ASSEMBLY: LONG-HORIZON COMPLEX MANIPULATION TASK

Furniture assembly is a hierarchical, long-horizon task requiring planning, perception, dexterous manipulation, and force control. FurnitureBench operationalizes this challenge through reproducible hardware, printable models, standardized initialization, demonstrations, and reproducibility testing.

  • Diverse dexterous skills: Furniture assembly combines high-level ordering decisions with accurate alignment, collision-aware path planning, reliable 3D perception, and deliberate force control.
  • Long-horizon tasks: Furniture assembly requires selecting parts, grasping, moving, aligning attachment points, and inserting or screwing them across repeated assembly steps.The task averages 60–230 seconds and 600–2300 low-level steps.
  • The benchmark focuses on single-arm manipulation, using a fixed table obstacle and modified furniture models to compensate for limited ability to handle multiple or large objects.Extension to multi-arm and mobile manipulation is left for future work.
  • Reproducible benchmark environment: FurnitureBench provides reproducible real-robot infrastructure, including a plug-and-play control stack, 3D-printable furniture models, setup guidance, and demonstration data.The system uses widely available hardware and printable models to support replication across laboratories.
  • Reproducibility analysis: 84% average performance was achieved in 10 newly built environments relative to the original environment, supporting benchmark reproducibility.Performance across new environments remained within 75–93% of the original, although policies were sensitive to camera viewpoints and backgrounds.
  • Reproducible benchmark environment: The task-initialization tool samples configurations from a predefined distribution and guides users to match furniture-part poses, improving comparability across evaluations.

B. Easy-to-Use Benchmark

FurnitureBench lowers the barrier to reproducible real-world experiments through standardized control, initialization, demonstrations, and simulation tools. FurnitureSim supports faster experimentation while preserving similar simulation-to-real trends.

  • A plug-and-play control stack, task initialization tool, and successful-demonstration dataset reduce the human effort required to use the benchmark.The initialization tool samples configurations from a predefined distribution, improving comparability across users.
  • A Docker-based robot control stack built on Polymetis lowers the entry barrier for real-robot manipulation experiments.
  • 219.6 hours of teleoperated successful demonstrations support offline RL and imitation learning on long-horizon assembly tasks.Demonstrations use an Oculus Quest 2 controller and keyboard, including fine-grained wrist rotation for screwing.
  • FurnitureSim provides a fast simulated version of FurnitureBench for quick testing and debugging, with realistic physics and rendering.It is built on Isaac Gym and Factory, including simulation of screws and realistic rendering.
  • Simulation and real-world results show similar trends across dataset sizes and algorithm choices on one_leg assembly.The comparison reports both success rate and completed phases.

V. EXPERIMENTAL SETUP

The experimental setup evaluates BC and IQL under controlled initialization conditions using success rate and completed phases. It separately measures individual assembly skills and acknowledges that TAMP is outside the paper’s main evaluation scope.

  • Baselines: BC learns demonstration state-action pairs by supervised learning, whereas IQL combines advantage-weighted BC with an expectile-regression value function.
  • Evaluation procedure: Evaluation records success rate and completed phases over ten episodes after standardized low, medium, or high-randomness initialization.Trials stop after unsafe motion, 350 steps per skill, or 3000 total steps.
  • Single-skill benchmark: Single-skill policies are trained from manually segmented trajectories and evaluated independently from skill-specific initial states.The benchmark evaluates the first five skills of each furniture model.
  • Single-skill benchmark: Grasping and placing are learned successfully, while inserting reaches only 0–20% success and screwing varies sharply by furniture model and geometry.IQL reaches 90% and 70% on screwing for square_table and desk, but 10% and 0% for lamp and round_table.
  • Scope: The paper benchmarks end-to-end learning approaches and leaves engineering TAMP procedures without state information outside its scope.

B. Benchmark II: Full-Assembly Benchmark

Full-assembly evaluation shows that current BC and IQL policies make little progress on the complete tasks, while the easier one_leg task provides a measurable testbed. Performance improves with easier initialization and more same-distribution data, and wrist views are important.

  • Full assembly: Neither BC nor IQL completes a single part assembly on full-assembly tasks, although IQL consistently outperforms BC and lower initialization randomness improves performance.
  • One_leg assembly: IQL-R3M achieves 4 completed phases on average and 40% success on low-randomness one_leg assembly.It always reaches phase 3, fails insertion 60% of the time, and usually screws successfully after insertion.
  • Data scaling: 2000 mixed demonstrations raise IQL-R3M performance to 4.6 and 3.7 completed phases on low- and medium-randomness settings, respectively.
  • Data diversity: Training on low-randomness data yields 3.8 completed phases versus 3.0 from medium-randomness data, indicating weak use of diverse data distributions.
  • Camera observations: Removing the wrist camera reduces completed phases from 3.8 and 3.0 to 2.0 and 1.3 on low- and medium-randomness settings.Policies mostly fail to grasp the table leg without wrist-camera observations.
  • Interpretation: The benchmark remains tractable but requires substantial data, while policies mainly fail at difficult inserting and screwing phases.

C. Benchmark III: Simulation Benchmark

FurnitureSim provides a faster experimental proxy for FurnitureBench, but simulation-to-real transfer remains limited by visual and physical gaps. The benchmark also remains bounded by simplified models, single-task evaluation, and single-arm manipulation.

  • Simulator role: FurnitureSim can accelerate testing and debugging, although direct knowledge transfer to the real world is difficult because of simulation-to-real domain gaps.
  • Simulation results: In simulated one_leg assembly, IQL achieves higher success rates than BC across visual encoders, while BC has comparable average completed phases.
  • Model scope: The furniture models use pieces wider than 2 cm and smaller overall than real-world furniture, simplifying grasping and workspace demands.The authors state that common challenges such as perception, exploration, temporal credit assignment, and dexterous manipulation remain represented.
  • Evaluation scope: The benchmark is mainly single-task and single-robot, leaving multi-task learning and multiembodiment evaluation for future work.
  • Simulation limitation: FurnitureSim has visual and physical simulation-to-real gaps, including mismatched robot trajectories caused by inaccurate mass and joint-friction modeling.The authors identify system identification as especially challenging.

APPENDIX A OVERVIEW

FurnitureBench combines widely available hardware, a reproducible real-robot setup, and a simulator for evaluating complex assembly algorithms. Its system separates client-side sensing and policy inference from server-side real-time robot control while using OSC to convert low-frequency policy actions into joint torques.

  • Reproducible system design: FurnitureBench uses a Franka Emika Panda, three Intel RealSense D435 cameras, a TOMMARYD table, and 3D-printed workspace objects.The setup also uses widely available products and a green backdrop for a consistent workspace.
  • System architecture: The server performs real-time robot control, while the client handles camera and robot inputs, policy inference, and action commands.This client-server division is illustrated in the system overview.
  • FurnitureSim: FurnitureSim replaces real-world interactions with PhysX contact simulation while reusing FurnitureBench code and 3D furniture models.The simulator is designed for fast evaluation while minimizing simulation-to-real gaps.
  • FurnitureSim: FurnitureSim estimates 3D-model density from measured mass and volume and uses a friction coefficient of 0.15 for experiments.The authors report that coefficients between 0.15 and 0.42 work robustly across diverse furniture configurations.

D. Reward Function

FurnitureBench defines sparse assembly rewards from relative part poses and supplements them with phase-based progress evaluation. Its policy interface uses visual and proprioceptive observations, while initialization and pose estimation procedures support consistent evaluation.

  • Reward function: +1 is awarded when a pair of parts is assembled, so a successful task receives total reward N −1 for N parts.Each pair is rewarded only once; disassembling and reassembling does not produce another reward.
  • Reward function: Assembly success is determined by comparing AprilTag-estimated relative poses with predefined ground-truth poses and thresholds.The supplied passage specifies rotation-matrix cosine similarity and absolute-distance criteria.
  • Phase evaluation: The phase score counts completed subtasks and provides fine-grained evaluation because the first reward can require many environmental steps.Human operators judge phase completion during evaluation, and phase information is not used for training.
  • Action and observation spaces: The policy uses an 8D action space comprising delta position, delta quaternion orientation, and gripper action bounded between −1 and +1.Binary gripper commands are activated only when the policy output exceeds an absolute threshold of 0.019.
  • Action and observation spaces: The policy observes front and wrist RGB images together with robot proprioceptive states.Available proprioception includes end-effector pose, velocity, and gripper width.
  • Pose estimation: Two-camera and multi-marker AprilTag processing improves furniture-pose estimation by aggregating estimates and filtering outliers.Marker placeholders and canonical part poses support consistent marker placement and pose computation.
  • Initialization and reproducibility: Three initialization randomness levels perturb initial furniture poses, while a GUI samples configurations and guides users to reproduce them.This standardizes initial-state distributions across evaluations.

APPENDIX E DATASET DETAILS

The dataset covers eight 3D-modeled furniture tasks with extensive teleoperated demonstrations, structured annotations, and varied assembly challenges. Qualitative evaluations show that BC and IQL often complete early pick-and-place phases but struggle with insertion and other precise interactions.

  • Dataset collection: 219 hours of demonstrations were collected over eight furniture models from two human operators.Collection used 100–250 demonstrations per model at low and medium randomness and 50 per model at high randomness.
  • Dataset contents: Each demonstration timestep contains RGB-D frames from three cameras, robot actions, rewards, AprilTag poses, and metadata.Low- and medium-randomness demonstrations also include phase-completion masks.
  • Dataset variation: The benchmark varies lighting and front-camera pose to cover diverse configurations, while high-randomness phases remain unannotated because assembly orders vary across trajectories.Human operators corrected erroneous automatic rewards during collection.
  • Furniture tasks: The eight furniture models introduce distinct interactions, including slipping, collision avoidance, rail alignment, insertion, flipping, and screwing.The models are modified for one-handed assembly with a single robotic arm.
  • Furniture models: The models are designed for 3D printing, with largest dimensions below 21 cm and printing times of approximately 10 to 24 hours.Some small or thin parts are thickened to provide space for AprilTag markers.
  • Qualitative results: Qualitative rollouts show BC and IQL generally completing initial pick-and-place phases but struggling with insertion.Examples include IQL failing tabletop-leg insertion and BC failing tabletop placement on square_table.
  • Qualitative results: The qualitative results include failures involving small grasping regions and repeated screwing after tightening.These examples span lamp, square_table, and round_table tasks.

B. Mount Robot on Table

The setup guide specifies precise robot, backdrop, AprilTag, and camera placement procedures to support reproducible FurnitureBench environments. It covers two front-camera mounting options and detailed cable and alignment checks.

  • Robot placement: The robot is installed 34.5 cm from the table’s left edge and tightly attached without margin.The guide also requires the mount to be tightly screwed.
  • Background installation: A green backdrop should minimize wrinkles and shadows, cover the table’s narrow side, and fill at least one-third of both table edges.These conditions are intended to standardize the front-camera background.
  • Base AprilTag: The base AprilTag establishes the world coordinate system, so its position, orientation, and flat attachment must be precise.The guide places its center at 24.5 cm horizontally and 37.5 cm vertically, with less than 2 mm error recommended.
  • Camera installation: The system uses front, rear, and wrist cameras whose serial numbers must be recorded before installation.The front camera is aimed at the base AprilTag, while the rear camera is mounted beside the robot base with collision clearance.
  • Camera installation: The front camera can use either a Ulanzi mount positioned 8 cm from the table edge or a Manfrotto mount positioned 7 cm away.Both options are described as reliable, with later fine-grained calibration available.
  • Camera installation: The wrist camera is mounted on the end-effector’s rear side and aimed toward the gripper tip.The guide also requires tightened screws, cable ties, and additional cable slack.

G. Install Software

The setup installs separate client and server software stacks, configures network and camera connections, and calibrates the robot, cameras, and workspace before operation.

  • Software installation: The client requires FCI 4.2.2, nvidia-docker2, the FurnitureBench code and Docker image, environment variables, and the client container.The client is launched with ./launch_client.sh --gpu --pulled.
  • Software installation: The server requires a real-time kernel, the FurnitureBench code and Docker image, server environment variables, and the server daemon.The server daemon is launched with /furniture-bench/launch_daemon.sh.
  • Connection setup: Server, client, and robot communicate through a local Ethernet network, with server and camera addresses or serial numbers configured through client-container environment variables.Camera variables include CAM_WRIST_SERIAL and CAM_FRONT_SERIAL; the rear-camera serial is also required.
  • Camera setup: Camera checks require wrist, front, and rear views in order, both gripper tips visible in the wrist view, and two rear-camera markers detectable on the base tag.The camera checklist is run with python furniture_bench/scripts/run_cam_april.py.
  • Robot operation: Robot operation requires unlocking the robot, releasing the activation button, activating FCI, launching the server daemon, and resetting the robot from the client terminal.The reset command is python furniture_bench/scripts/reset.py.
  • Calibration: Front-camera calibration overlays the current view on a pre-recorded reference and uses pose deviations to guide alignment.Position deviations must remain within ±0.004 and rotation deviations within ±0.8; acceptable values appear green.

K. Install Obstacle

Obstacle installation uses a calibration GUI and double-sided rubber tape, followed by lighting, calibration, software, and initialization checks before policy evaluation.

  • Install Obstacle: The obstacle pose is calibrated with the provided GUI before attachment to the table.The calibration tool is run with python furniture_bench/scripts/calibration.py --target obstacle.
  • Install Obstacle: The obstacle must match the transparent reference pose without discrepancy and be firmly secured with double-sided rubber tape so it does not move when pushed.The attachment procedure is illustrated in Figures 35 and 36.
  • Evaluation setup: Evaluation lighting should remain similar to the specified setup, with the light on the table’s left side, 4600 K-6000 K temperature, and 500 lm-1000 lm brightness.Data collection randomizes lighting, whereas evaluation requires stable conditions.
  • Evaluation setup: A pre-trained one_leg policy test checks the environment by running through tabletop pickup, corner pushing, leg pickup, insertion, and screwing phases.The expected assessment is completion of over 3 phases on average.
  • Evaluation setup: Before evaluation, operators recheck calibration, minimize backdrop wrinkles, clean camera lenses, install evaluation requirements, randomize furniture placement, and run the one_leg evaluation script.The evaluation command is ./evaluate.sh --low --one_leg.
  • Task initialization: After GUI initialization reports that furniture parts are correctly aligned, pressing Enter executes the policy with only furniture parts in the workspace.Random placement precedes initialization and evaluation.
Loading 2305.12821v1…