Source-linked AI summary
RAMP: A Benchmark for Evaluating Robotic Assembly Manipulation and Planning
Jack Collins, Mark Robson, Jun Yamada, Mohan Sridharan, Karol Janik, Ingmar Posner
TL;DR
Researchers seeking to engage only in particular sub-tasks aligned with their interests face significant barriers to entry. RAMP is an accessible, challenge-driven, open-ended assembly benchmark with public parts, simulation, instrumented components, and baselines for focused or full-system work. 84% mean success and 580sec mean time summarize the baseline across the three easy assemblies.
Problem
Researchers seeking to engage only in particular sub-tasks aligned with their interests face significant barriers to entry.
Source: §I · p. 2Method
RAMP is an accessible, challenge-driven, open-ended assembly benchmark with public parts, simulation, instrumented components, and baselines for focused or full-system work.
Source: §I · p. 2Results
84% mean success and 580sec mean time summarize the baseline across the three easy assemblies.
Source: C. Results · p. 6Takeaways & Limitations
RAMP’s reconfigurable parts and planned extensions support a community-driven benchmark that can evolve as capabilities mature.
Source: §I · p. 2, §VI · p. 7
Abstract
from arXiv · showhide
We introduce RAMP, an open-source robotics benchmark inspired by real-world industrial assembly tasks. RAMP consists of beams that a robot must assemble into specified goal configurations using pegs as fasteners. As such, it assesses planning and execution capabilities, and poses challenges in perception, reasoning, manipulation, diagnostics, fault recovery, and goal parsing. RAMP has been designed to be accessible and extensible. Parts are either 3D printed or otherwise constructed from materials that are readily obtainable. The design of parts and detailed instructions are publicly available. In order to broaden community engagement, RAMP incorporates fixtures such as April Tags which enable researchers to focus on individual sub-tasks of the assembly challenge if desired. We provide a full digital twin as well as rudimentary baselines to enable rapid progress. Our vision is for RAMP to form the substrate for a community-driven endeavour that evolves as capability matures.
I. INTRODUCTION
RAMP targets the manual, variable assembly stage of offsite construction, where automation requires capabilities spanning perception, planning, execution, and recovery. It is designed as an accessible, challenge-driven, open-ended benchmark that supports both full-system work and focused subtask research.
I. INTRODUCTION
- Offsite beam assembly remains manual and varies between projects, demanding perception, long-horizon planning, robust execution, and fault recovery.The beams can be cut and punched automatically, but assembling them into frames remains manual because assemblies are often different.Source: §I · p. 1, Figure 2 · p. 2
- Robotics benchmarks face barriers from heterogeneous hardware, unequal resources, and the tension between evaluating full systems and enabling focused subtask research.Robotic assembly requires perception, planning, and control, while researchers may want to address only particular sub-tasks.Source: §I · pp. 1–2
- RAMP provides easy, medium, and hard assemblies with an evaluation protocol and metrics promoting repeatable, fast, and complete assembly.The benchmark is designed to reflect offsite construction challenges.Source: §I · p. 2
- Open-source parts, detailed instructions, an accurate simulation, instrumented components, and real-world and simulation baselines support access and focused subtask work.The components use widely available 3D-printed parts and extruded aluminium profiles.Source: §I · p. 2
- RAMP is intended to grow through reconfigurable parts, new configurations, dual-arm setups, and possible tactile-sensing and deformable-object extensions.The authors envision a community-driven effort that evolves as capabilities mature.Source: §I · p. 2
II. LITERATURE REVIEW
Existing robotic assembly benchmarks and competitions provide useful task settings, but many are difficult to access or do not assess long-horizon planning and generalization. RAMP combines accessible, reconfigurable parts with physical multi-step assembly evaluation.
II. LITERATURE REVIEW
- Many assembly benchmarks assess short-horizon tasks or simulation alone; only a subset evaluate long-horizon success and real-world performance.Methods assessed on unique, hard-to-access tasks are difficult to compare directly.Source: §II · p. 2
- NIST assembly boards cover varied manipulation challenges, but simpler boards need no sequencing and their many distinct parts hinder accessibility and expansion.The boards use fixed goal configurations.Source: §II · p. 2
- The bimanual semi-deformable benchmark assesses fine-motor interactions, but its sequential single-goal tasks omit high-level planning and require bimanual hardware.Its tasks include watchmaking-inspired assembly and rubber-band manipulation.Source: §II · pp. 2–3
- Competitions offer common tasks and often shared hardware, but parts are not readily accessible to non-competitors, leaving a need for an open-source alternative.Source: §II · p. 3
- RAMP addresses gaps in accessibility and fixed-goal evaluation with easy-to-create, reconfigurable parts and physical long-horizon, multi-step reasoning tasks.The same parts can support many designs and assessment of generalization to new goal configurations.Source: §II · p. 3
III. BENCHMARK
RAMP jointly assesses robotic assembly planning and execution through an accessible, open-ended benchmark designed around challenging tasks. Its current scope is single-arm manipulation, with components for comparable evaluation and simulation-based prototyping.
III. BENCHMARK
- RAMP jointly assesses planning and execution for robotic assembly, with challenge-driven, accessible, and open-ended design principles.Source: §III · p. 3
- The benchmark currently targets single-arm manipulation and includes predefined beams, assemblies, performance metrics, and a reproducibility-oriented evaluation protocol.Source: §III · p. 3
- An accurate simulation environment supports prototyping and testing alongside the benchmark’s physical assembly tasks.Source: §III · p. 3
A. Base Parts
RAMP combines accessible, extensible physical parts with tiered assembly goals that demand increasingly complex skills and planning. Goal configurations are provided as images and XML, and serve as a holdout set for related designs.
A. Base Parts
- Nine base beams and up to 15 pegs use extruded aluminium profiles and 3D-printed joints, with part files, a bill of parts, and an assembly guide available.Selected joints have space for 25×25mm April Tags to assist state estimation and beam identification.Source: A. Base Parts · p. 3
- Additional printable fingertips, clamps, and peg holders, plus an A2 layout template, support manipulation, localization, and reproducible setup.The benchmark anticipates evolving its beams and assemblies toward more challenging designs.Source: A. Base Parts · p. 3
B. Tasks
- Easy assemblies require 3–4 peg insertions; medium tasks add angled insertion, sliding, and re-grasping, while hard free-form designs require longer action sequences and added planning.Source: B. Tasks · p. 3
- Goal states are supplied as images and XML beam configurations; the XML format supports existing planners and can be generated automatically.Other goal modalities, including natural language, are possible and encouraged.Source: B. Tasks · p. 3
- The three task classes pose open problems in perception, state estimation, planning, control, execution, fault recovery, and learning; similar designs can serve as holdouts.Source: B. Tasks · p. 3
C. Performance Measures
RAMP measures accurate completion and time, and defines a repeatable evaluation protocol. The setup supports different robot and sensor configurations while fixing the starting beam and parts.
C. Performance Measures
- Each assembly class is evaluated through completion accuracy and time, counting a peg only when it correctly and fully pins the required joints.Time includes generating the executed plan; results include averages across five repeats and the most successful attempt.Source: C. Performance Measures · pp. 3–4
- The benchmark groups goal configurations into three assembly classes.The figure presents the classes for assessing benchmark capabilities.Source: Figure 3 · p. 4
D. Evaluation Protocol
- The protocol holds code constant across a class and requires five consecutive runs per assembly, changing only the goal.This is recommended to support reproducibility and assess robustness and repeatability.Source: D. Evaluation Protocol · p. 4
- RAMP is intended for manipulators with state-estimation sensors and encourages diverse sensor setups and grippers, from parallel grippers to anthropomorphic hands.The first beam is fixed within the robot’s operational space, and the other parts begin in allocated spots on the A2 template.Source: D. Evaluation Protocol · p. 4
E. Simulation
RAMP provides a simulation environment intended to mirror its real-world setup, with physics and observations for developing and assessing solutions. The paper also introduces a baseline as a starting point for new methods.
E. Simulation
- The Nvidia Isaac simulation uses Signed Distance Fields for fine-resolution contact detection, including joint and peg insertion.The environment is designed to mirror the real-world setup.Source: E. Simulation · p. 4, Figure 4 · p. 5
- The simulator provides state, force-torque, and wrist and birds-eye camera observations, plus interfaces for Cartesian position and impedance control.It enables development and assessment without access to a real-world setup.Source: E. Simulation · p. 4
- The paper identifies observation noise, April Tags, additional control methods, procedural generation, and parallelization as future simulation extensions.The proposed additions include force-based and closed-loop policies.Source: E. Simulation · p. 4
IV. BASELINE IMPLEMENTATION
- The baseline is intended to lower the barrier for new adopters and provide a foundation for new methods.It uses a task planner conditioned on an XML-specified goal assembly and hand-engineered skills to execute the plan.Source: §IV · pp. 4–5
A. Task Planner
The baseline planner adapts a refinement-based architecture to reason over coarse and fine domain descriptions. It plans abstract actions and maps each transition to executable fine-resolution actions.
A. Task Planner
- The planner adapts REBA, which reasons over domain transition diagrams at coarse and fine resolutions, with the fine description formally refining the coarse one.REBA supports inference, planning, and diagnostics amid noisy sensing and actuation.Source: A. Task Planner · p. 5
- The coarse description includes domain sorts, actions, statics, and fluents, with axioms for causal laws, state constraints, and executability conditions.Examples include putdown causing an object to leave the robot’s hand and making pickup impossible when the object is already held.Source: A. Task Planner · p. 5
- ASP encodes the coarse domain, initial conditions, prior observations and executions, and search heuristics to compute abstract action sequences for a goal.The baseline uses SPARC and the clingo SAT solver to compute answer sets.Source: A. Task Planner · p. 5
- For each abstract transition, a refined fine-resolution description and bridge axioms support planning a sequence of fine actions for execution.The listed actions include moving, picking up, putting down, assembling, fastening, and pushing.Source: A. Task Planner · p. 5
B. Action Execution
The baseline executes hand-designed skills in open loop, using camera-based April Tag observations and motion or impedance control. Its easy-class results are plotted as completion against planning and execution time.
B. Action Execution
- The ordered fine-resolution plan runs open loop through hand-designed skills, while cameras observe April Tags to estimate beam and peg locations.Known offsets from tags to beam joints and links allow their poses to be computed in the global frame.Source: B. Action Execution · pp. 5–6
- Fig. 5 plots completion percentage against planning and execution time for three easy assemblies, showing average, best, and standard deviation across repeats.The caption notes that standard deviation increases with execution time because more complex assemblies typically require more actions.Source: Figure 5 · p. 6
- MoveIt handles simple motions, while Cartesian impedance control and force feedback support contact-rich actions such as fastening.For peg insertion, the baseline detects contact, searches in a spiral for the hole, and retries after re-estimating its position if needed.Source: B. Action Execution · p. 6
- Beam assembly uses force feedback to detect contact and align joints; capping is followed by pushing to correct beam position.The push action aims to restore nominal alignment for beams inserted later.Source: B. Action Execution · p. 6
C. Results
The baseline averages 84% success and 580 seconds across the three easy assemblies, with planning taking 180–200 seconds. Peg insertion and alignment failures limit performance, especially as assembly action counts grow.
C. Results
- 84% success and 580 seconds are the baseline’s average mean success and mean time across the three easy assemblies.Planning takes 180–200 seconds.Source: C. Results · p. 6
- Peg insertion failures often stem from noisy hole estimates, slanted gripped pegs, or imprecise beam alignment.The authors identify insertion as a difficult benchmark action.Source: C. Results · p. 6
- The standard deviation increases with time, indicating lower success rates for assemblies requiring more actions.Source: C. Results · p. 6
- Figure 6 visualizes the baseline robot’s attempted plan for the third easy assembly.Source: Figure 6 · p. 7
V. LOOKING AHEAD
The authors identify baseline improvements and benchmark extensions spanning fault recovery, manipulation skills, sensing, and new assembly settings. These directions include diagnostics and replanning, tactile sensing, and multiagent or 3D tasks.
V. LOOKING AHEAD
- Adding diagnostics to the task planner could support fault detection and replanning, while tighter task–motion planner integration could improve skill-sequence generation and execution.Source: §V · p. 7
- Learned motion plans, learned contact-rich manipulation, and added sensing such as tactile input are proposed avenues for improving skill robustness and speed.Source: §V · p. 7
- The single-arm benchmark could be extended to multiagent or dual-arm coordination, disassembly, and 3D assembly, each introducing additional task challenges.Source: §V · p. 7
- Removing fiducial markers would add object identification and pose estimation challenges while making the benchmark more representative of real-world problems.Source: §V · p. 7
VI. CONCLUSION
RAMP is presented as a challenge-driven, accessible, and open-ended robotics benchmark with open-source physical components and a real-world and simulation baseline. The authors envision further baseline development and benchmark expansion as a community-driven effort.
VI. CONCLUSION
- RAMP combines open-source, easy-to-source parts with a rudimentary baseline implemented on both real-world hardware and in simulation.Source: §VI · p. 7
- The reconfigurable parts support an open-ended range of assembly designs, and adopters can build a full system or extend the provided baseline.Source: §VI · p. 7
- The authors plan to develop the baseline further and expand RAMP to settings such as bimanual manipulation as a community-driven effort.Source: §VI · p. 7