Source-linked AI summary
VirTooS: A ROS 2 - Unity Virtualization Toolkit for Fleet Management of Autonomous Mobile Robots
Andrea Drudi, Lorenzo Pichierri, Andrea Testa, Giuseppe Notarstefano
TL;DR
VIRTOOS targets the lack of a ROS 2–Unity toolkit for mixed-reality fleet management of AMRs and distributed coordination. It combines modular ROS 2 and Python components with C# Unity integration, virtualization, sensors, and containerized deployment. Experiments demonstrate task assignment and interaction between real and virtual robots.
Problem
Existing approaches do not provide a ROS 2–Unity toolkit explicitly tailored to mixed-reality fleet management of AMRs executing distributed coordination algorithms.
Method
VIRTOOS combines ROS 2 and Python fleet-management components with C# Unity integration, modular layers, virtual sensing, and containerized deployment.
Results
Experiments demonstrate virtual task servicing, real-robot navigation in a virtual environment, and mixed-reality interaction between a physical robot and virtual robots.
Takeaways & Limitations
VIRTOOS provides a platform for designing custom virtual environments, implementing cooperative algorithms, and evaluating multi-robot coordination when complete physical fleets are unavailable.
Abstract
from arXiv · showhide
In this paper, we present VirTooS, a Python/C# toolkit designed to implement fleet-management tasks on teams of Autonomous Mobile Robots (AMRs). VirTooS leverages the Robot Operating System (ROS) 2 and Unity game engine to provide realistic, scalable virtual experiments in a mixed-reality environment. The toolbox allows users to easily generate and customize virtual scenarios for realistic simulations. Virtual and real sensors as, e.g., LiDARs, can be exploited to map and safely navigate in the mixed-reality environment. To enable distributed robotics experiments, we propose a set of tailored routines leveraging the ChoiRbot framework. As a motivating example, we show a set of experiments for task assignment problems in a virtual environment, allowing seamless interaction among real and virtual robots. Moreover, the package comes with a containerized suite to easily deploy it on different machines. The source code will be made publicly available on GitHub.
I. INTRODUCTION
VIRTOOS addresses the lack of a ROS 2–Unity toolkit tailored to mixed-reality fleet management of AMRs executing distributed coordination algorithms. It combines ROS 2, Unity, modular software, and containerization for customizable virtual and mixed-reality experiments.
- Existing robotics toolboxes cover ROS 2 multi-robot systems, domain-specific applications, and simulation, but none is explicitly tailored to mixed-reality AMR fleet management.
- VIRTOOS is a ROS 2–Unity toolkit for implementing fleet-management tasks on teams of autonomous mobile robots in virtual and mixed-reality environments.
- The toolkit uses modular robot components for high-level decision-making, trajectory generation, planning, control, and exchange of data with Unity.
- VIRTOOS supports photorealistic virtual environments whose sensor measurements reflect Unity, enabling operations such as mapping in complex indoor environments.
- The paper is organized around the VIRTOOS architecture, the Unity–Gazebo Virtualization Interface, and virtual and real experiments.
II. ARCHITECTURE OVERVIEW
VIRTOOS couples three software layers to support complex cooperative robotic scenarios: team management, single-robot autonomy, and Unity-based virtualization.
- The toolbox divides into an optimization-based team management system, a single-robot control, navigation, and planning system, and a Unity-based interface.
- Coupling the layers lets users design optimization algorithms, control laws, and virtual environments for cooperative robotic scenarios.
- The Unity interface supports testing and validation of cooperative strategies in realistic virtual environments.
A. Optimization-based Management System
The architecture separates fleet-level decision-making from single-robot autonomy and adapts established ROS 2 navigation and SLAM packages to shared multi-robot environments.
- Optimization-based Management System: Fleet behavior uses a decision-making layer that assigns tasks from overall system state and mission objectives, with distributed approaches motivated by growing task and fleet complexity.
- Single-Robot Control, Navigation and Planning System: The single-robot layer converts high-level management decisions into suitable control inputs for individual robots.
- Single-Robot Control, Navigation and Planning System: VIRTOOS adapts Nav2 routines to multi-robot scenarios, supporting local and global replanning and navigation in a shared environment.
- Single-Robot Control, Navigation and Planning System: Multi-robot extensions to SLAM Toolbox support shared-environment SLAM without overlapping ROS 2 routines.
C. Unity – Gazebo Virtualization Interface
The Unity–Gazebo interface combines Unity’s virtual environment with Gazebo physics and exchanges sensor data and control commands with ROS 2 to integrate real and virtual robots.
- Unity generates indoor environments, robots, and objects, while Gazebo simulates robot dynamics as the physics engine.
- The interface exchanges sensor data and control commands between robots and both virtual and real environments.
- The communication scheme connects Unity and ROS 2 for mixed-reality experiments with configurable laser scanner sensors.
III. UNITY-BASED INTERFACE FOR VIRTUAL SENSING AND MIXED-REALITY SIMULATIONS
The Unity-based interface provides a realistic, interactive simulation front-end for ROS-controlled multi-robot systems. It supports customizable virtual environments and robot configurations for cooperative testing and validation.
- Unity-based simulation interface: Unity provides realistic 3D environments where robots can navigate and interact with objects in real time.Its graphics and physics capabilities support dynamic, interactive multi-robot simulations.
- Unity-based simulation interface: GameObjects, Prefabs, and Scripts represent scene entities, reusable robotic and environmental assets, and custom simulated behaviors.Scripts also support communication between Unity and ROS 2.
- Unity-based simulation interface: Unity serves as an interactive simulation front-end while ROS manages sensor handling, path planning, and control algorithms.This division allows complex environments and robotic behaviors to be simulated under ROS control.
- Dynamic generation of virtual environments: A parametric routine customizes indoor layouts, walls, shelves, and lighting through configuration parameters.Floor tiles use an adjustable matrix pattern, while objects are placed on a grid aligned with the ROS 2 reference frame.
- Dynamic generation of virtual environments: The Unity Robot menu allows users to personalize robot settings for the simulation.
- Dynamic generation of virtual environments: Unity can generate a 30 m × 30 m indoor environment in which 50 Jackal ground robots navigate.
2) Robot Integration in Unity:
Robot integration combines imported real-world models, coordinate transformations, message exchange, and configurable laser scanners. This connects Unity virtual robots with ROS 2 and Gazebo for mixed-reality experiments.
- Robot Integration in Unity: The Jackal model is imported into Unity from a URDF file using Unity’s URDF Importer.
- Robot Integration in Unity: Custom scripts reconcile Unity’s RUF and Gazebo’s FLU coordinate conventions while exchanging poses and virtual sensor data.
- Robot Integration in Unity: The Unity interface provides configuration controls for laser scanner parameters.
IV. MIXED-REALITY EXPERIMENTS
VIRTOOS experiments evaluate virtual and mixed-reality mobile-robot tasks in a virtualized indoor facility. The setup uses ROS 2 communication and containerized software deployment.
- Mixed-reality experiments: The experiments include three scenarios involving virtual and/or real mobile robots performing tasks in a virtualized indoor facility.
- Experimental setup: The Jackal communicates with the workstation through ROS 2 publisher-subscriber messaging over a standard 5 GHz Wi-Fi network.
- Experimental setup: The real and virtual experiments are dockerized and run on Ubuntu 24.04 with ROS 2 Jazzy Jalisco and Unity 6000.0.74f1.
A. Scenario 1 – Dynamic Task Assignment in Virtual Indoor Environments with Obstacles
The first scenario applies distributed Dynamic Task Assignment to four virtual Jackal robots navigating an indoor environment. Nav2, laser scanning, SLAM, and Unity–Gazebo integration support task execution, mapping, and collision avoidance.
- Scenario 1 – Dynamic Task Assignment: The robots cooperatively solve a dynamically evolving task-assignment problem through distributed optimization.Each robot determines its task assignment before the single-robot system manages navigation and planning.
- Scenario 1 – Dynamic Task Assignment: N = 4 Clearpath Jackal robots navigate a virtual indoor environment using Nav2 for control and local replanning from laser scanner measurements.
- Scenario 1 – Dynamic Task Assignment: Custom Unity scripts generate laser scanner measurements, while Gazebo forwards pose information.
- Scenario 1 – Dynamic Task Assignment: The experiments show robots servicing tasks in settings with randomly placed shelves and shelves arranged in racks.
- Scenario 1 – Dynamic Task Assignment: A Jackal navigates in Unity while mapping its surroundings and avoiding collisions with obstacles and other robots.
B. Scenario 2 – Real Robot in Virtual Environment
This experiment demonstrates a physical Jackal navigating while sensing a Unity-built virtual environment. The robot is tasked with reaching virtual goal poses among virtual obstacles.
- Scenario setup: A physical Jackal navigates in the laboratory while sensing a virtual environment built with the Unity–Gazebo Virtualization Interface.The robot pose is tracked by a Vicon MoCap system and continuously updated in real time.
- Navigation task: The robot reaches a series of goal poses within the virtual environment in the presence of virtual obstacles.This task highlights the mixed-reality interconnection between the real robot and the virtual environment.
C. Scenario 3 – Mixed-Reality Experiment
This mixed-reality experiment combines one physical Jackal with three virtual robots to solve a Dynamic Task Assignment problem. The physical robot interacts with the virtual fleet while navigating safely in a constrained room.
- Scenario setup: N = 4 Jackal robots cooperatively service tasks, with one robot operating physically and three operating in the virtual environment.The physical robot is mirrored in the virtual world for interaction with the virtual environment.
- Scenario setup: The experiment uses a 4 m × 8 m room to facilitate physical-robot navigation in the laboratory.Robot1,R is the physical robot in this scenario.
- Mixed-reality navigation: Robot1,R navigates toward its assigned goal while avoiding static obstacles and virtual robots through the shared mixed-reality framework.The physical robot uses the local navigation stack while the fleet management layer provides high-level task allocation.
- Mixed-reality navigation: Between 20 s and 30 s, limited motion occurs because local collision avoidance prevents safe progress in the constrained environment.Navigation resumes after the fleet management layer updates target assignments.
- Other applications: VIRTOOS can support cooperative mapping, localization, distributed optimization, vision-based navigation, and coordination in exploration applications.The paper identifies applications including vehicle routing, human–robot interaction, and space robotics.
V. CONCLUSIONS
VIRTOOS is a Python and C# toolkit for fleet-management tasks involving mobile robots in complex virtual and mixed-reality environments. Its modular, containerized architecture combines ROS 2 and Unity with virtual sensing and established navigation tools.
- Conclusions: VIRTOOS enables high-fidelity virtual experiments in mixed-reality scenarios using ROS 2 and Unity.The toolkit provides interfaces for custom environments, cooperative algorithms, and virtual onboard sensors.
- Conclusions: Its three software layers cover optimization-based team management, single-robot control and navigation, and Unity-based virtual sensing and simulation.The modular structure supports designing custom optimization algorithms, control laws, and virtual environments.
- Conclusions: A containerized architecture supports deployment across diverse robotic platforms, while Nav2 and SLAM Toolbox extend the toolkit’s applicability.These integrations provide established navigation, control, and mapping capabilities within the fleet-management framework.