Source-linked AI summary
Baxter's Homunculus: Virtual Reality Spaces for Teleoperation in Manufacturing
Jeffrey I Lipton, Aidan J Fay, Daniela Rus
TL;DR
Expensive specialized systems have limited the development of manufacturing telerobotics. This paper demonstrates a consumer-VR-based teleoperation architecture built around a virtual control room and adaptable mappings, achieving 100% success versus 66% for an automated system while supporting multiple task types.
Problem
Expensive specialized systems have hampered development of telerobotic systems for manufacturing.
Method
The system embeds the user in a virtual reality control room using consumer VR hardware, gaming software, and dynamic mappings between human inputs and robot state.
Results
The system achieved 100% success in 20 pick-and-stack trials versus 66% for a state-of-the-art automated algorithm and supported pick-and-place, assembly, and manufacturing tasks.
Takeaways & Limitations
The architecture is flexible across network architectures and bandwidth allotments while leveraging existing consumer hardware and software to operate existing robots remotely.
Abstract
from arXiv · showhide
Expensive specialized systems have hampered development of telerobotic systems for manufacturing systems. In this paper we demonstrate a telerobotic system which can reduce the cost of such system by leveraging commercial virtual reality(VR) technology and integrating it with existing robotics control software. The system runs on a commercial gaming engine using off the shelf VR hardware. This system can be deployed on multiple network architectures from a wired local network to a wireless network connection over the Internet. The system is based on the homunculus model of mind wherein we embed the user in a virtual reality control room. The control room allows for multiple sensor display, dynamic mapping between the user and robot, does not require the production of duals for the robot, or its environment. The control room is mapped to a space inside the robot to provide a sense of co-location within the robot. We compared our system with state of the art automation algorithms for assembly tasks, showing a 100% success rate for our system compared with a 66% success rate for automated systems. We demonstrate that our system can be used for pick and place, assembly, and manufacturing tasks.
I. INTRODUCTION
The paper presents a VR-based telerobotic system for manufacturing that combines a virtual control room, consumer hardware, and adaptable human–robot mappings. Demonstrations compare its assembly performance with state-of-the-art automation and extend it to multiple manufacturing tasks.
- I. INTRODUCTION: The virtual reality control room decouples user and robot inputs and outputs while providing adjustable mappings between them.It uses commercial VR systems and gaming infrastructure for manufacturing tasks.
- I. INTRODUCTION: The architecture supports telerobotic operation across wired local, building-wide wireless, and hotel wireless connections from another city.These deployments demonstrate operation over different network architectures.
- I. INTRODUCTION: 100% success was achieved in 20 pick-and-stack trials, compared with 66% for a state-of-the-art automated algorithm.The system had an average task time of 52 seconds and comparable speed to state-of-the-art In-hand Object Localization systems.
- I. INTRODUCTION: Demonstrations covered pick-and-place, fixture-less assembly, and wire manipulation using a commercial staple gun.The system also handled screws, flexible materials, and complex shapes.
- I. INTRODUCTION: The system applies the homunculus model to teleoperation, enabling virtual co-location between the user and robot.The user is situated in a control room mapped to a space inside the robotic system.
II. BACKGROUND
Teleoperation depends on mappings between user, robot, sensor, and gripper states, but existing piloting, mimicking, and cyber-physical approaches impose latency, bodily incongruities, modeling, or data requirements.
- Teleoperation mappings: Teleoperation maps user-space hand and eye states to robot grippers and sensors despite differences in degrees of freedom, joint ranges, and state counts.Positions and orientations are represented with reference frames, object identifiers, and item indices.
- Homunculus model: The Homunculus model uses a virtual space to decouple human and robot mappings, allowing different sensor and gripper mappings and user-adjustable MHG control.It aims to preserve virtual co-location while transmitting less data and avoiding discomfort associated with piloting and mimicking.
- Teleoperation mappings: Unexpected delays or relative motions between proprioception and vision can cause simulator sickness, making rapid eye- and hand-movement mapping important.The paper links proprioception to feeling present in virtual reality and incongruent feedback to nausea and motion sickness.
- Existing approaches: Piloting systems provide camera data through conventional controls, while mimicking systems directly map user and robot states to create co-location.Mimicking can fail when stationary cameras require the user to hold their head still, causing fatigue and nausea.
- Existing approaches: Direct body-movement mapping can be limited by mismatched arm joints, degrees of freedom, sizes, and ranges, reducing comfort or workspace utilization.The paper notes that Baxter can reach positions or contortions that are painful or impossible for a human user.
- Existing approaches: Cyber-physical systems maintain virtual robot, environment, and user duals, but require complete models and networked updates.Shared virtual spaces support training and repetitive-task teaching, yet the cited approach lacks a human in the controls for changing situations and requires substantial environment data or prior knowledge.
III. DESIGN OF VR ENVIRONMENTS FOR THE HOMUNCULUS MODEL
The VR environment places the operator in a virtual control room mapped inside the robot, separating user interaction from robot state while integrating stereo vision, sensors, and adjustable gripper mappings.
- Virtual control room: The VR control room maps human space into virtual space and then robot space, while local rendering rapidly updates user stimuli independently of robot motion and sensor loops.Virtual cameras compensate for head movement, and the system targets the frame-rate requirements of VR applications.
- Virtual control room: The design avoids a complete virtual robot dual, instead showing controls and markers for the robot’s last known position and orientation.This decouples the robot-to-virtual mapping into MGG and MDS.
- Spatial mappings: λDS >> λGG scales displayed camera objects larger than the user because of stereo-camera parallax and virtual-space scaling.The paper defines λGG as a scale factor within the coordinate transform and applies a larger display-space scale for the camera view.
- Gripper control: Users selectively couple their hands to virtual grippers for position and orientation, then release them so a planner can generate robot joint commands.The control algorithm lets users choose position-and-orientation or position-only mappings and colors an invalid solution as an error.
- Visual interaction: Stereo views let the user’s brain extract 3D data without reconstructing a physical-environment dual, reducing streamed and processed data.The VR control room combines a stereo work-space window with virtual cameras that respond to user movement.
- Visual interaction: The interface combines each arm’s video and range-sensor display with a central stereoscopic view, supporting whole-system or single-arm attention shifts.Users can compare current location markers and commanded endpoints for both arms simultaneously.
IV. THE HOMUNCULUS MODEL SYSTEM
The system uses consumer VR hardware and a virtual control room to present robot sensors and support flexible user-to-robot control mappings. Its interface combines multiple views, virtual controls, and networked communication for teleoperation.
- Hardware: The system uses an Oculus Rift, touch controllers, and a Baxter robot equipped with arm sensors and configurable end-effectors.Baxter was fitted with parallel plate grippers or a gripper and staple gun, while stereo cameras provided head-mounted visual input.
- Sensor displays: Each arm’s hand-camera video and range-sensor value are displayed alongside a stereoscopic view of the arm.This lets the user combine overall stereo vision with close hand-camera and range information.
- Virtual controls: Users control robot arms through virtual glowing orbs and markers showing commanded validity and current position and orientation.A virtual grid provides an additional reference surface in the control room.
- Adaptive mapping: The mapping between user controllers and robot arms can be changed for the task, including position-only control with fixed downward orientation.This transform makes downward-facing gripper operations more comfortable for pick-and-place tasks.
- Demonstrated manipulation: The interface supports grasping objects from multiple angles, transferring them between hands, and placing flexible or rigid objects into a bucket.The demonstrated objects include a cylinder, cloth, and reel, requiring dragging and coordinated transfer.
- System architecture: The VR control room separates user and robot inputs and outputs while mapping the virtual space into the robot’s space.The system uses Unity and LCM to connect consumer VR hardware on Windows with Baxter’s ROS and Ubuntu software.
V. RESULTS AND DISCUSSION
The system was evaluated on pick-and-place, block assembly, fixture-less assembly, wire stapling, and multiple network configurations. It achieved perfect assembly performance in the reported comparison and remained operable over wireless Internet with reduced update rates.
- Pick-and-place: The system evaluated pick-and-place operations using objects with different shapes and compliance, requiring grasp selection, hand transfer, re-grasping, and bin placement.The evaluation included eight different objects and leveraged user intuition about object responses.
- Block assembly: A user combined stereo and hand cameras to approach, grasp, align, and assemble stacked blocks.The block assembly task tested system performance through coordinated visual guidance and manipulation.
- Manufacturing task: The system enabled a user to place a wire with one arm, align a staple gun with the other, and secure the wire to a board.Without the hand camera, the user needed 5 attempts to secure 2 staples, but stereo depth information still supported alignment.
- Network configurations: The robot was controlled over a local wired network, same-building wireless networks, and hotel Internet connections.The long-range test used 1Hz hand-camera updates, 5Hz range updates, 800x600 stereo resolution, and a 5 Mbps connection cap.
- Fixture-less assembly: Fixture-less assembly required switching between multiple MHG mappings for table grasping, free-space assembly, and handoffs.Rotation was fixed during table grasping and directly mapped during free-space assembly or transfers.
VI. CONCLUSION
The paper concludes that homunculus-model VR telerobotics can operate across network conditions while supporting flexible mappings and consumer hardware. The architecture is presented as a path toward broader remote operation of existing robots.
- Conclusion: The developed VR telerobotics system is flexible across network architectures and bandwidth allotments.Its human-to-robot and robot-to-human mappings can be changed to improve utility and ergonomics.
- Conclusion: Consumer-grade hardware and software frameworks allow the system to remotely operate existing robot systems and support wider deployment.The conclusion connects this capability to remote work in manufacturing environments and at scales users cannot currently reach.