Source-linked AI summary
Programming and execution of skill-based human-robot-crane collaborative tasks
Taneli Lohi, Markku Suomalainen, Roope Mellanen, Tapio Heikkilä
TL;DR
Highly varying manufacturing and indoor logistics require robotic systems with greater agility, flexibility, and robustness. The paper presents a CAD-model-based system for programming and executing parameterized skills and monitors through an event-based modified Behavior Tree, demonstrated in human-robot-crane handling of heavy components.
Problem
Robotic manufacturing and indoor logistics must accommodate highly varying products, product mixes, volumes, material flows, and quality conditions while including cranes and human operators.
Method
The paper uses CAD-model-based programming of parameterized robot skills and monitors within a dynamic, event-based modified Behavior Tree architecture that includes cranes and human operators.
Results
The system was tested in a human-robot-crane assembly task that moved a heavy load to a target object for attachment by a human operator.
Takeaways & Limitations
Skill-based workflows with programmable monitors provide the paper's demonstrated approach to flexible and robust robot-task execution, including nonrobotic resources.
Abstract
from arXiv · showhide
Highly varying production sets increasing challenges for robotic manufacturing and indoor logistics. New capabilities for agility, flexibility, and robustness are needed. Robot skills, integrating motions, tool operations, and sensor perceptions consistently provide an execution mechanism for a versatile set of tasks with varying parameters. In this paper, easy-to-use CAD-model based programming and execution system for parametrized skills and skill monitors is showcased. The execution control structure is dynamic and parametrized, based on a modified Behavior Tree, where only event based communication is used. A human-robot-crane collaborative skill is shown as a test example, where a human instructs an overhead crane and a manipulator in inserting a heavy object supported by the crane, and guided by the manipulator, into the goal.
I. INTRODUCTION
Highly variable manufacturing and indoor logistics require agile, flexible, and robust robotic systems that can accommodate heavy machinery and human operators. The paper introduces an integrated system for programming and executing human-robot-crane collaborative tasks using skills and Behavior Tree control.
- Manufacturing automation must handle frequent product, product-mix, volume, material-flow, and quality variations.
- Robot skills integrate motions, tool operations, and sensor perceptions to execute versatile tasks with varying parameters.
- Production environments combine robots with overhead cranes, industrial PLC-controlled machinery, and human operators.
- The proposed system provides integrated programming and execution for human-robot-crane collaboration, using Behavior Trees, parallel skill execution, failure reaction, and monitoring skills.
A. Skill based robotics
Skill-based robotics encapsulates perception, decision-making, and control into parameterized capabilities organized across task, skill, and primitive layers. Related systems address programming, monitoring, and runtime adaptation, while this approach emphasizes CAD-based programming and inclusion of cranes and human operators.
- Robot skills encapsulate perception, decision-making, and control for specified tasks, then support reuse through parameterization and sequencing.
- Existing work covers object-centric, uncertainty-aware, constraint-based, knowledge-based, and finite-state-machine approaches to skill programming and monitoring.
- The proposed architecture extends skill-based systems by supporting CAD-based skill sequencing and integrating nonrobotic resources such as commercial cranes and human operators.
- Execution monitoring helps robot systems prepare for and react to failing executions by detecting and indicating behavioral anomalies.
- Monitoring approaches include analytical, data-driven, and knowledge-based methods, with the latter including causal analysis, expert systems, and neural networks.
C. Behavior Trees
Behavior Trees provide reusable control structures for task switching, runtime restructuring, and recovery from unforeseen situations. The paper adopts event-based execution and modifies parallel-node status reporting to make behavior more reactive.
- Behavior Trees support adaptation to unforeseen situations, error recovery, and operation in dynamic environments.
- Behavior Tree nodes are reusable components, unlike finite-state-machine nodes that depend on other nodes.
- The related event-based approach triggers tree processing when nodes finish or shared blackboard state changes, reducing processing when nothing happens.
III. ARCHITECTURE
The architecture organizes robot control into task, skill, and primitive layers, with parametrized components selected from a skill library and sequenced through CAD-based programming. Skills use primitives for physical interaction, while monitors support task-level recovery.
- Three-layer architecture: The architecture follows task, skill, and primitive layers, with parametrized skills and primitives implemented as reusable skill-library components.
- Task layer: Tasks are created by an operator or programmer using a modified Behavior Tree, with skills selected from the library.
- Skill and primitive layers: Skills are parameterized by control-recipe values and world-model queries, then interact with the physical world through atomic primitives.
- Three-layer architecture: The control layers communicate through a hierarchy in which skills and skill monitors are Behavior Tree leaf nodes and primitives support environmental interaction.
B. Modified Behavior Tree
The system uses a Behavior Tree to order skills and monitors, with event-based execution and parallel nodes that can react immediately to failures.
- B. Modified Behavior Tree: Behavior Trees organize task control through sequence, parallel, fallback, execution, and condition nodes, with skills and monitors as execution leaves.The root node has one child, while control nodes can have multiple children and compose nested control structures.
- B. Modified Behavior Tree: The modified Behavior Tree replaces tick-based execution with event-based communication and changes parallel-node status logic for more reactive behavior.These changes are intended to react instantly to failure events.
- B. Modified Behavior Tree: The world model stores workspace objects and cell configuration, supports get-method queries, and receives object poses from perception primitives.Entities provide a base representation that can be extended with attributes for different entity types.
- B. Modified Behavior Tree: A modified parallel node starts all children simultaneously, then reports success or failure after user-defined thresholds of child outcomes and shuts down remaining active children.The thresholds M and K determine how many successful or failed child events trigger the parent result.
IV. MODELING AND IMPLEMENTATION OF TASKS AND SKILLS
Tasks combine a Behavior Tree with world-model checks, while skills and primitives are modeled as synchronized control sequences across activity-model lifelines.
- IV. MODELING AND IMPLEMENTATION OF TASKS AND SKILLS: Skill, primitive, and task control flows are modeled with UML activity diagrams, whose swim lanes represent skill or primitive lifelines.The models express control flow and synchronization between skills and primitive operations.
- IV. MODELING AND IMPLEMENTATION OF TASKS AND SKILLS: The task-layer activity is omitted from skill-model design and included only for illustration in the activity diagram.
- IV. MODELING AND IMPLEMENTATION OF TASKS AND SKILLS: The GUI represents skills with robot-arm symbols, monitors with gauge symbols, Behavior Tree control nodes with arrows, and conditions with condition nodes.These elements form the visual task representation described for Figure 3.
- IV. MODELING AND IMPLEMENTATION OF TASKS AND SKILLS: The task layer contains the world model and Behavior Tree, which can check preconditions such as object localization before starting skills.Skills query object poses from the world model, while monitors use primitives as data sources.
A. CAD based planning of task and skills
CAD-based planning lets operators build Behavior Tree task recipes from library components, parameterize skills using model features, and execute them through a task controller.
- A. CAD based planning of task and skills: The CAD planner lets operators select model faces, edges, and vertices, then build a control tree by connecting library skills and monitors to objects.The system also supports compatibility checks between skills.
- A. CAD based planning of task and skills: Skill parameters are set by selecting CAD features, such as a point and perpendicular edges for a pose, or by entering data fields manually.
- A. CAD based planning of task and skills: The task controller is a Python ROS2 node that reads a JSON control recipe and creates and executes the corresponding Behavior Tree.The recipe is passed from planning software to the robot control system as a file or through TCP/IP.
- A. CAD based planning of task and skills: Runtime data is stored in a world model inside the root node, allowing skill executors to query localized workspace objects during execution.The world model is populated by the Localize skill and updated by handling skills.
- A. CAD based planning of task and skills: Skills use predefined control sequences modeled with UML activity models and implemented as finite-state machines, while executors communicate with parent or root nodes through events, messages, and queries.Primitive operations can be implemented as Python functions, C++ executables, or robot-language program blocks, with ROS2 services and topics for asynchronous operations.
A. Human-robot-crane control
The test demonstrates collaborative crane-and-robot handling of a heavy load, combining monitored grasping, visual guidance, and human-directed crane motion in an assembly scenario.
- A. Human-robot-crane control: The experiment moves a heavy load to a target using an overhead crane and lightweight robot, after which a human attaches the load with screws.The collaboration removes the need for the operator to push the load while simultaneously controlling the crane.
- A. Human-robot-crane control: The “Attach robot to load” flow combines a grasp primitive with a parallel “Detect yank” monitor that checks whether grasping succeeds.
- A. Human-robot-crane control: The load can be controlled through human joystick instructions or a 3D visual-servo controller that tracks the six-dimensional poses of the load and target.Both robot and crane motions use admittance controllers that transform contact force into velocity-control signals.
- A. Human-robot-crane control: The “Insert visually guided” skill coordinates its primitive operations to support visually guided insertion of the load into the target.The skill and primitive operations are shown as synchronized control flows.
- A. Human-robot-crane control: The setup used a Konecranes CXT Smart crane with a 3200 kg maximum load, alongside a KUKA LBR iiwa14 robot, magnetic gripper, and 3D camera.A KUKA Quantec KR 210 was also used as a crane emulator during initial skill testing.
- A. Human-robot-crane control: The task was planned as a sequence of skills and primitives, with CAD models generating grasp poses, visual-servo waypoints, and screw locations.The programmer entered additional parameters such as yank-monitor thresholds, screw types, and target torques before the JSON recipe was executed.
VI. DISCUSSION
The system combines skill-based task programming with Behavior Tree control, programmable monitors, and CAD-derived parameters for flexible human-robot-crane collaboration. A test sequence demonstrates coordinated crane, robot, and operator actions in heavy-object assembly.
- Control and monitoring: Behavior Tree control supports parallel skill and monitor execution, anomaly detection, and fallback-based corrective operations.Fallbacks can invert skills, activate alternatives, or defer decisions to an operator.
- Control and monitoring: The control hierarchy organizes the assembly task through tasks, skills, monitors, and primitive operations.Skills are implemented through manually defined programs and state-transition models derived from UML Activity models.
- Programming and parameterization: CAD models generate parameters for skills, while programmers can provide additional parameters directly.Recipe scripts configure skill and monitor parameters using CAD model data.
- Human-robot-crane test: The test sequence coordinates an overhead crane carrying the load, a robot pushing it to an attachment position, and an operator fastening it with screws.A screw prompt identifies the target hole and provides screw type and target torque.
- Programming and parameterization: Perception skills, precondition checks, and execution monitors address environmental uncertainty and inaccuracies during task execution.The system is agnostic to skill origins, allowing learned parameters or newly learned skills and primitive operations.
VII. CONCLUSIONS
The paper introduces a skill-based system for easy programming and flexible execution of robot tasks. It combines perception and robot operations into workflows, with Behavior Tree control and programmable monitors supporting flexibility and robustness.
- Robot activities are modeled as skills that integrate perceptions and robot operations into purposeful workflow sequences.
- Behavior Tree control over skills and programmable monitors provide the system’s flexibility and robustness mechanisms.