Source-linked AI summary
Unity: A General Platform for Intelligent Agents
Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, Danny Lange
TL;DR
Existing AI environments often lack realistic sensory and physical properties, complex tasks, social interaction, or flexible configuration. The paper proposes a taxonomy of simulation platforms, examines Unity and Unity ML-Agents as a general platform, and surveys research they support. It concludes that modern game engines offer a natural direction for richer, more flexible AI environments, while existing platforms remain insufficient for long-term progress.
Problem
Many research environments provide limited visual, physical, task, social, or configurational complexity, restricting the environments available for AI research.
Method
The paper proposes a taxonomy of simulation platforms and analyzes Unity with the open-source Unity ML-Agents Toolkit as a general platform.
Results
Unity ML-Agents supports configurable environments with complex observations, interactions, and benchmark tasks, including Unity environments used for challenging research.
Takeaways & Limitations
Modern game engines are presented as a natural next step for creating flexible environments with richer visual, physical, task, and social complexity.
Takeaways & Limitations
Existing platforms such as the Arcade Learning Environment and Project Malmo remain constrained by black-box configuration, limited realism, or restricted environment logic.
Abstract
from arXiv · showhide
Recent advances in artificial intelligence have been driven by the presence of increasingly realistic and complex simulated environments. However, many of the existing environments provide either unrealistic visuals, inaccurate physics, low task complexity, restricted agent perspective, or a limited capacity for interaction among artificial agents. Furthermore, many platforms lack the ability to flexibly configure the simulation, making the simulated environment a black-box from the perspective of the learning system. In this work, we propose a novel taxonomy of existing simulation platforms and discuss the highest level class of general platforms which enable the development of learning environments that are rich in visual, physical, task, and social complexity. We argue that modern game engines are uniquely suited to act as general platforms and as a case study examine the Unity engine and open source Unity ML-Agents Toolkit. We then survey the research enabled by Unity and the Unity ML-Agents Toolkit, discussing the kinds of research a flexible, interactive and easily configurable general platform can facilitate.
1. Introduction
The paper argues that increasingly sophisticated AI research requires richer, more flexible simulation environments. It proposes a taxonomy of platforms and examines Unity with ML-Agents as a general platform for creating such environments.
- Simulation environments are foundational for testing reinforcement learning ideas, yet their design has received less discussion than algorithms.
- As algorithms solve existing benchmarks, new environments are needed to sustain meaningful challenges and drive further algorithm development.
- Modern game engines offer realistic visuals, sophisticated physics, complex agent interactions, and intuitive interfaces for creating challenging simulations.
- The paper proposes a taxonomy classifying research platforms by sensory, physical, task-logic, and social complexity.
- Unity and the open-source Unity ML-Agents Toolkit are analyzed as examples of a general platform, including their architecture, tools, and benchmark environments.
2. Anatomy of Environments and Simulators
The paper characterizes useful learning environments by their sensory, physical, task, and social complexity, alongside practical requirements for flexible, fast, distributed simulation. These properties determine how richly and efficiently researchers can study intelligent behavior.
- 2.1 Environment Properties: Environmental complexity must increase as algorithms improve, particularly across sensory, physical, task-logic, and social dimensions.
- Sensory Complexity: Sensory complexity supports learning from rich visual, auditory, and textual information relevant to real-world decision-making.
- Physical Complexity: Physical complexity requires realistic dynamics and control interactions when agents operate in environments modeled on the real world.
- Task Logic Complexity: Task-logic complexity can arise from large search spaces, hierarchical structure, varying instances, sequential tasks, and continual learning requirements.
- Social Complexity: Social complexity concerns learning behavior among groups of agents and relationships within social settings.
- 2.2 Simulation Properties: Simulators must be flexibly controlled and run quickly and in parallel because modern methods may require billions of samples.Fast, distributed simulation increases data collection and shortens experimental iteration.
3. A Survey of Existing Simulators
The survey organizes simulators by flexibility, from fixed black-box environments to general platforms capable of supporting arbitrarily complex visuals, physics, tasks, and social interactions. Existing platforms provide useful capabilities but remain constrained by their underlying engines, rendering, physics, perspectives, or scenario logic.
- Taxonomy: Four categories organize simulators by environment-specification flexibility: Environment, Environment Suite, Domain-specific Platform, and General Platform.The first two provide fixed environments or packaged benchmark sets; domain-specific platforms create tasks within a restricted domain.
- Taxonomy: General Platforms can create environments with arbitrarily complex visuals, physical and social interactions, and tasks, encompassing the other categories.The paper identifies modern game engines, paired with research-specific abstractions and interfaces, as strong candidates for this category.
- Arcade Learning Environment: The ALE provides pixel rendering, hierarchical task logic, and emulation at thousands of frames per second, but its environments are deterministic, immutable, and black-box.The passage also reports relatively simple visuals, limited physics, single-agent control, and greater-than-human performance across its environments.
- Physics Simulators: MuJoCo offers high-quality physics and standardized continuous-control benchmarks but limits visual rendering and dynamic game-like environments.Its compiled models make real-time instantiation and destruction of many objects more difficult, restricting tasks requiring greater planning or coordination.
- Game-based Platforms: VizDoom supports first-person navigation and control, yet its decades-old engine limits visual and physical complexity and restricts agents to a first-person perspective.Project Malmo similarly supports flexible scenarios and multi-agent research, but Minecraft constrains visuals, physics, and scenario logic.
4. The Unity Platform
Unity combines real-time 3D rendering, physics, and an editor designed for flexible content creation across platforms. Its engine and services support learning environments spanning simple gridworlds to complex multi-agent games, along with custom sensing, demonstrations, and distributed simulation.
- Platform Overview: Unity combines a rendering engine, physics engine, and graphical Unity Editor for creating interactive 2D, 3D, and AR/VR simulations.The platform is used across gaming, architecture, engineering, construction, automotive, and film applications.
- Engine Properties: Unity’s flexible engine supports tasks ranging from simple 2D gridworlds to complex 3D strategy games, physics-based puzzles, and multi-agent competitive games.This breadth follows from Unity’s general-purpose design for varied platforms, developer experience levels, and game types.
- Sensory Complexity: Unity enables high-fidelity rendering with baked or real-time lighting, custom shaders, and outputs such as depth, object masks, infrared, and noisy images.These capabilities can produce near-photorealistic imagery and varied sensor data for machine-learning training.
- Task and Social Complexity: Unity’s component system supports multiple agents, policies, and environments, enabling hierarchical or meta-learning tasks and straightforward multi-agent scenarios.Its scripting and component abstractions also support social complexity inherited from multiplayer-game development.
- Tools and Simulation: Asynchronous physics and rendering allow faster simulation, including non-rendered execution when visuals are unnecessary.The Editor also supports custom Scenes, local expert demonstrations, and deployment across more than 20 platforms for large-scale demonstrations or human-performance measurement.
5. The Unity ML-Agents Toolkit
The Unity ML-Agents Toolkit provides an open-source framework for creating configurable Unity learning environments and interfacing with them through Python. Its SDK supports varied agent observations, actions, rewards, policies, algorithms, multi-agent setups, runtime environment changes, and benchmark examples.
- Toolkit Overview: The open-source toolkit enables researchers to create Unity environments in the Unity Editor and interact with them through a Python API.Its ML-Agents SDK includes functionality for defining environments and core C# scripts for learning pipelines.
- Algorithms and Extensions: Supported learning methods include SAC, PPO, GAIL, Behavioral Cloning, Self-Play, ICM extensions, and LSTM extensions.Self-Play is supported in symmetric and asymmetric games.
- Agents and Policies: Agents can collect rendered-image, ray-cast, or variable-length vector observations, take actions, and receive rewards through configurable policies.Agents may use discrete or continuous actions and dense or sparse rewards.
- Agents and Policies: The toolkit supports multiple agents sharing behavior policies, multiple behavior types, and decision mechanisms including player input, scripts, and neural networks.This enables multi-agent scenarios with groups or individual agents executing different behavior types.
- Environment Control: The Academy coordinates simulation steps and agents while runtime parameters can alter physics, textures, sizes, and GameObject existence.These controls support train/test environment variations and curriculum-learning scenarios.
- Example Environments: The toolkit includes example environments for benchmarking and as templates, spanning single- and multi-agent tasks with varied observations and action spaces.The examples include Basic, 3DBall, Crawler, Push Block, Tennis, Worm, Bouncer, Grid World, Walker, Reacher, and Food Collector.
6. Research Using Unity and the Unity ML-Agents Toolkit
Research using Unity and the Unity ML-Agents Toolkit spans varied algorithms and environments, demonstrating the platform’s flexibility. Obstacle Tower illustrates the complexity and challenge achievable with Unity, while its competition stimulated diverse algorithmic solutions.
- Related Platforms: Unity-based platforms also include AI2-THOR for first-person indoor navigation and Arena benchmarks focused on multi-agent scenarios.AI2-THOR research demonstrated transfer from simulation to a physical robot for indoor navigation.
- Research Applications: Unity environments have supported research on intrinsic motivation, neural attention, semi-parametric reinforcement learning, and morphological self-assembly.The self-assembly work trained individual agents to achieve higher-order tasks such as standing or locomotion.
- Obstacle Tower: The Obstacle Tower benchmark uses procedural generation, sparse rewards, and 100 randomly generated floors with increasingly complex layouts.These design choices require flexible decision-making across varied task instances.
- Obstacle Tower: Obstacle Tower’s open competition evaluated agents on five held-out towers, and the top entry solved an average of 19 floors.This exceeded naive human-level performance but remained below expert human play and optimal performance.
- Obstacle Tower: The top-performing competition approach combined reinforcement learning, imitation learning, human demonstrations, and state augmentations.The paper presents this combination as an example of how novel environments can promote novel algorithms.
7. Potential for Future AI Research
The paper identifies research directions that current platforms make difficult, including evolving environments, human-guided learning, and agents trained alongside humans. Unity’s programmability, editor, and browser deployment are presented as ways to support these directions.
- 7.1 Effective Learning Environments: Procedurally generated and evolving environments are important for studying generalization, robustness, and artificial general intelligence.Unity supports parameterization, procedural generation, changing dynamics, and dynamic creation or destruction of entities.
- 7.2 Human-in-the-loop Training: Current platforms make human evaluative feedback and direct manipulation of agent states or actions difficult or impossible.This limitation contrasts with imitation learning, which requires comparatively little additional platform functionality.
- 7.2 Human-in-the-loop Training: Unity’s visual editor enables interactive human-agent training through pausing, moving or changing objects, and assuming control of the agent.These interventions are intended to simplify feedback administration and environment modification during training.
- 7.3 Training Agents Alongside Humans: Training agents to challenge humans without discouraging future play may require objectives beyond maximizing episode return.The paper discusses optimizing both expected return and the number of expected future episodes.
- 7.3 Training Agents Alongside Humans: Unity’s WebGL deployment enables agent-human interaction studies at scale through browser-based games.The paper also notes that training against humans with varied play styles can improve policy generalization and robustness.
8. Conclusion and Future Directions
The paper proposes general platforms as a framework for evaluating simulators, argues that current platforms are insufficient for long-term progress, and examines Unity as a flexible alternative. It identifies future work in usability, performance, and breadth.
- 8. Conclusion and Future Directions: The paper introduces general platforms for environment creation and analyzes Unity with the Unity ML-Agents Toolkit as an example.It also discusses desirable simulator complexity and computational properties.
- 8. Conclusion and Future Directions: A novel taxonomy is used to argue that current platforms are insufficient for long-term progress and that modern game engines are a natural next step.The conclusion also highlights Unity’s role in spurring innovation and enabling research that inflexible platforms make burdensome.
- 8. Conclusion and Future Directions: Future work includes intuitive interfaces for non-experts to tune rewards, observations, actions, and other algorithm-design choices.The paper identifies hyperparameter tuning as potentially insurmountable for some non-expert game developers.
- 8. Conclusion and Future Directions: Further development is planned to improve the Unity engine and Unity ML-Agents Toolkit in both performance and breadth.