Source-linked AI summary
CoSTAR: Instructing Collaborative Robots with Behavior Trees and Vision
Chris Paxton, Andrew Hundt, Felix Jonathan, Kelleher Guerin, Gregory D. Hager
TL;DR
CoSTAR addresses the challenge of enabling non-experts to program collaborative robots for varied tasks despite costly programming and limited human-meaningful perception. It provides a modular Behavior Tree system that integrates perception with abstract world representations, and evaluates the resulting task plans across industrial robots and increasingly complex tasks.
Problem
Existing robot-programming systems have clumsy interfaces and limited human-meaningful perception, while setup costs and environmental variation hinder broad deployment.
Method
CoSTAR combines a modular, cross-platform Behavior Tree task editor with perception-based symbols, predicates, object pose estimation, and abstract world representations.
Results
The system was evaluated on increasingly complex tasks across industrial robots, including repeatable assembly experiments and demonstrations on KUKA and UR5 platforms.
Takeaways & Limitations
Abstract perception and modular components support task plans designed to remain capable, usable, and robust to environmental variation.
Abstract
from arXiv · showhide
For collaborative robots to become useful, end users who are not robotics experts must be able to instruct them to perform a variety of tasks. With this goal in mind, we developed a system for end-user creation of robust task plans with a broad range of capabilities. CoSTAR: the Collaborative System for Task Automation and Recognition is our winning entry in the 2016 KUKA Innovation Award competition at the Hannover Messe trade show, which this year focused on Flexible Manufacturing. CoSTAR is unique in how it creates natural abstractions that use perception to represent the world in a way users can both understand and utilize to author capable and robust task plans. Our Behavior Tree-based task editor integrates high-level information from known object segmentation and pose estimation with spatial reasoning and robot actions to create robust task plans. We describe the cross-platform design and implementation of this system on multiple industrial robots and evaluate its suitability for a wide variety of use cases.
I. INTRODUCTION
CoSTAR addresses costly, difficult robot programming by combining a Behavior Tree interface with perception-based abstractions for capable, usable, and robust task plans across industrial platforms.
- Motivation: Existing robot-programming systems combine clumsy interfaces with limited human-meaningful perception, while setup, configuration, and environmental variation hinder deployment.These barriers especially affect small manufacturers that cannot easily absorb retooling and reprogramming costs.
- Design goals: CoSTAR targets capability, usability, and robustness so non-experts can create varied task plans that remain effective under reasonable environmental variation.The system is designed for repeated executions to produce expected results.
- System contribution: The system extends a Behavior Tree architecture into a modular, cross-platform framework integrating perception with an abstract world representation.The paper reports deployment on KUKA LBR iiwa and Universal Robots UR5 platforms.
- Authoring approach: CoSTAR combines a graphical interface with grounded sensor abstractions, requiring active user involvement while producing more robust and predictable task plans.The approach relies on users’ domain knowledge rather than requiring a complex ontology.
- Perception and authoring: Abstract perception exposes symbols and predicates that let users author task plans using object and position information rather than low-level sensor data.CoSTAR’s Predicator supports tasks integrating object detection and pose estimation in cluttered scenes.
II. RELATED WORK
Related work spans demonstration-based learning, high-level specification, and visual programming. CoSTAR builds on these directions with a Behavior Tree interface and perception-grounded abstractions for industrial task construction.
- End-user collaboration: Robot collaboration research includes human demonstrations segmented into primitive actions and deep reinforcement learning for training individual actions.These approaches represent one route to end-user collaboration and robot skill acquisition.
- High-level specification: High-level task specification uses domain languages such as PDDL, natural language, or ontologies to describe tasks and share robot knowledge.Examples include kitting and household chores supported by object models and action recipes.
- Visual programming: Visual interfaces let users construct industrial robot solutions directly, including Snap-based programming and hierarchical finite-state-machine task descriptions.CoSTAR expands a previously described Behavior Tree visual interface.
- Behavior Trees: Behavior Trees support complex manipulation and humanoid control while offering scalability and modularity comparable in power to hierarchical finite-state machines.These properties motivate their use for collaborative robot task plans.
- CoSTAR’s position: CoSTAR combines a powerful graphical interface with grounded sensor abstractions, exposing symbolic and qualitative information for human-term task formulation.Perception also enables tasks such as sorting and supports operation despite movement of objects and goals.
A. Software Architecture
CoSTAR uses modular components that expose abstract symbols, predicates, and operations to a Behavior Tree, allowing task plans to reason over world state while components remain extensible.
- A. Software Architecture: Components expose input/output data, symbols, predicates, and operations, with Perception, Gripper, Arm, and Predicator forming the execution architecture.Additional components can be added when needed.
- A. Software Architecture: The Behavior Tree sees abstracted symbols, predicates, and operations rather than continuous component inputs and outputs.This abstraction lets components be modified or extended without directly changing the task plan.
- A. Software Architecture: Symbols represent objects and positions, while predicates describe their qualities and relationships or map inputs and symbols to boolean values.Predicates discretize continuous input into meaningful subsets for generalizable task plans.
- A. Software Architecture: Operations change stored symbols, robot state, or the physical world and typically appear as Behavior Tree leaf nodes.Move is given as an example operation.
- A. Software Architecture: Predicator aggregates component-produced symbols and predicates and exposes operations that let Behavior Trees query them.Its sub-components can be activated or deactivated according to system needs.
- A. Software Architecture: The parallel C-Model gripper supports only PinchMode, so attempts to use its other gripper-mode operations fail.The adaptive gripper supports BasicMode, PinchMode, WideMode, and ScissorMode.
B. User Experience
CoSTAR combines kinesthetic teaching with visual Behavior Tree authoring so users can create hierarchical task plans through demonstration and explicit instruction.
- B. User Experience: Users can physically teach trajectories or skill models and can author hierarchical Behavior Trees through a graphical interface.The system presents these as two complementary interaction methods.
- B. User Experience: Behavior Tree sequence, selection, repeat, and reset nodes compose complex task plans from operations and control execution flow.Sequence nodes stop on failure, while selection nodes stop after success.
- B. User Experience: Hand-eye calibration uses an arm-mounted AR marker and a Calibrate button so users can calibrate a robot to a camera when the marker is visible.CoSTAR uses dual quaternion hand-eye calibration to compute and save the marker transform.
- B. User Experience: The interface presents detected objects, SmartMoves, waypoints, a task workspace, operation lists, and menus for visual task development.SmartMove creation and other operation-customization panes are also provided.
- B. User Experience: Perception training is a system-setup procedure performed separately for each new object.The training algorithm is presented as a straightforward setup step.
IV. PERCEPTION
CoSTAR integrates RGB-D perception, object segmentation, pose estimation, and spatial predicates into an abstract representation that Behavior Trees can use for task authoring.
- IV. PERCEPTION: CoSTAR exposes high-level perceptual knowledge so users can author task plans robust to variations in object positions, orientations, and manipulation goals.The central integration challenge is making perception useful and understandable in an interactive programming environment.
- IV. PERCEPTION: The perception pipeline consumes RGB-D data, segments objects, estimates poses, and creates symbols for detected objects before Predicator produces predicates.The pipeline combines an AR marker tracker with object classification and ObjRecRANSAC-based pose estimation.
- IV. PERCEPTION: Waypoints are 6-DOF coordinate frames supplied by robot kinematics, AR markers, or object detection and pose estimation.The current perception-generated coordinate frames come from ObjRecRANSAC.
- IV. PERCEPTION: Predicates describe object classes and spatial relationships such as LeftOf and RightOf for perception-produced frames.Most symbols represent positions in Cartesian or joint space.
A. Object Pose Estimation Pipeline
The object pose-estimation pipeline addresses adaptation, pose symmetries, identity persistence, and Behavior Tree integration to support robust task-plan authoring.
- A. Object Pose Estimation Pipeline: The pipeline must adapt to new objects and environments, resolve rotational ambiguities, preserve object identities, and integrate perceptual operations with Behavior Trees.These requirements are stated as challenges for robust performance during task-plan authoring.
- A. Object Pose Estimation Pipeline: Median filtering, SVM segmentation, and pose estimation produce camera-frame coordinate estimates, while setCanonicalOrientation() resolves object symmetries.The function uses canonical orientation, prioritized axis ordering, and the physical object model.
- A. Object Pose Estimation Pipeline: persistenceUpdate() preserves identities across DetectObjects calls by matching same-class detections with nearby positions stored in an R*-Tree.New scene objects receive new names; the method is limited by update-rate velocity bounds and concave-object ambiguities.
B. Perception Setup
CoSTAR adapts perception to new environments through object-class models, symmetry information, and organized point-cloud data collected by non-experts.
- Deployment requires an SVM, 3D models for identified object classes, and symmetry information for those objects.
- The training procedure requires many organized point clouds showing multiple views of each object.
- Data collection uses an RGB-D camera and clear workspace, with box segmentation and clustering producing single-object training examples.
- Persistent identities are maintained by matching detected objects to nearby same-type entries within a maximum distance.
- Existing matches are removed before insertion, while unmatched detections receive unique identifiers before rebuilding the R*Tree.
C. Perceptual Operations
CoSTAR exposes perception through operations that update symbolic world knowledge and select feasible movement goals from predicate-qualified objects and poses.
- DetectObjects updates known object positions, orientations, types, waypoints, and associated predicates from RGB-D perception.
- DetectObjects creates explicit knowledge-update points so plans can require a non-occluded workspace view when perception runs.
- SmartMove generates candidate goals across detected objects and their symmetry transforms, then selects the minimum-cost goal for movement.
- SmartMove lets users select movement goals among same-class objects using predicates such as class and geometry.
V. EVALUATION
The evaluation examines CoSTAR through capability, usability, and robustness, demonstrating varied tasks and rapid plan construction across robot platforms.
- Capability asks whether the system can perform a particular task, usability asks how easily end users can adapt it, and robustness asks whether results repeat under variation.
- Tasks were constructed on-site in roughly 30 minutes despite differences between the LBR iiwa and UR5 platforms.
- CoSTAR plans were created for wire bending, surface polishing, and collaborative structure assembly on the LBR iiwa.
- The polishing demonstration used a complex Behavior Tree containing iterator, sequence, and reset nodes.
- The wire-bending case used blind programming because the wires were too small for the object-detection system to detect and localize.
Sanding and Polishing:
The demonstrations connect tool-based robot actions, behavior-tree control, spatial predicates, and perception to polishing and collaborative assembly tasks.
- Sanding and Polishing: Polishing tasks use Cartesian impedance movements to move a tool along a known path until interrupted.
- Sanding and Polishing: The polishing plan integrates external hardware through ToolOn and ToolOff operations and uses a selector to manage waiting and reset behavior.
- Collaborative Assembly: The assembly sequence adds a node, a connecting link, and another node to the structure using the UR5.
- Collaborative Assembly: The collaborative assembly task uses LeftOf and RightOf predicates to separate input materials from the assembly workspace.
- Perception-Based Tasks: Three perception-based tasks were created using sensor point clouds, object pose estimates, and robot joint states.
1. Pick up node:
The experiment progressively extended a node-manipulation task from picking up feasible nodes to assembling a structure with new objects and links. The final assembly succeeded in all 10 trials, despite one sensor-noise-induced pose inaccuracy.
- The UR5 first picked up any feasible node, then reused the task plan to move two nodes across the table.
- The user modified the program to place a new node, then added a connecting link and another node to the structure.
- 10 trials of final structure assembly succeeded in 10 out of 10 trials, with no perception failures.One pose inaccuracy caused by sensor noise was corrected during grasping, allowing successful placement.
- The system is intended to support varied, robust task plans, while formal usability assessment remained future work.