Source-linked AI summary
SPEAR: A Simulator for Photorealistic Embodied AI Research
Mike Roberts, Renhan Wang, Rushikesh Zawar, Rachith Dey-Prakash, Quentin Leboutet, Stephan R. Richter, Matthias Müller, German Ros, Rui Tang, Stefan Leutenegger, Yannick Hold-Geoffroy, Kalyan Sunkavalli, Vladlen Koltun
TL;DR
Existing Unreal Engine simulators offer limited programmability, integration, and rendering speed. SPEAR provides a Python-controlled, reflection-based simulator with expressive frame-level programming, exposing over 14K UE functions and rendering 1920×1080 images at 73 frames per second.
Problem
Existing UE-based simulators provide limited Python interfaces, incur high image-transfer overhead, and are difficult to integrate with existing projects or third-party assets.
Method
SPEAR uses a Python library, UE reflection, and transaction-based programming to dynamically control UE applications and execute data-dependent work within frames.
Results
73 frames per second at 1920×1080 is achieved for photorealistic beauty images, alongside over 14K exposed UE functions and diverse applications across UE projects.
Takeaways & Limitations
SPEAR supports a wider range of UE programs than existing simulators, including procedural content, path tracing, co-simulation, and natural-language scene editing.
Takeaways & Limitations
The programming model assumes Python and UE run in separate processes, with UE work executed eagerly through packaged SPEAR plugins.
Abstract
from arXiv · showhide
Interactive simulators have become powerful tools for training embodied agents and generating synthetic visual data, but existing photorealistic simulators suffer from limited generality, programmability, and rendering speed. We address these limitations by introducing SPEAR: A Simulator for Photorealistic Embodied AI Research. At its core, SPEAR is a Python library that can connect to, and programmatically control, any Unreal Engine (UE) application via a modular plugin architecture. SPEAR exposes over 14K unique UE functions to Python, representing an order-of-magnitude increase in programmable functionality over existing UE-based simulators. Additionally, a single SPEAR instance can render 1920x1080 photorealistic beauty images directly into a user's NumPy array at 73 frames per second - an order of magnitude faster than existing UE plugins - while also providing ground truth image modalities that are not available in any existing UE-based simulator (e.g., a non-diffuse intrinsic image decomposition, material IDs, and physically based shading parameters). Finally, SPEAR introduces an expressive high-level programming model that enables users to specify complex graphs of UE work with arbitrary data dependencies among work items, and to execute these graphs deterministically within a single UE frame. We demonstrate the utility of SPEAR through a diverse collection of example applications: controlling multiple embodied agents with distinct action spaces (e.g., humans, cars, and robots) across several in-the-wild UE projects; rendering photorealistic city-scale environments; manipulating UE's procedural content generation systems; rendering synchronized multi-view images of detailed human faces; coordinating an interactive co-simulation with the MuJoCo physics simulator; and editing scenes with natural language via an AI coding assistant.
1 Introduction
SPEAR is a Python-controlled Unreal Engine simulator designed to address existing UE-based simulators’ limited programmability, communication overhead, and integration constraints. It combines runtime reflection, direct NumPy image transfer, rich ground-truth modalities, and deterministic high-level UE work graphs, demonstrated across diverse embodied-AI applications.
- Core contribution: SPEAR connects to and programmatically controls any Unreal Engine application through a modular plugin architecture, exposing over 14K unique UE functions.This represents an order-of-magnitude increase in programmable functionality over existing simulators.
- Rendering: 73 frames per second: SPEAR renders 1920×1080 photorealistic beauty images directly into a user’s NumPy array, an order of magnitude faster than existing UE plugins.Its camera sensor also provides ground-truth image modalities unavailable in existing UE-based simulators.
- Programming model: SPEAR’s runtime-reflection interface lets Python dynamically find UE classes, call functions, and manipulate object variables using strings as keys.The interface is implemented through a comprehensive C++ layer interacting directly with UE’s runtime reflection system.
- Programming model: SPEAR provides native-Python and asynchronous variants for visible UE functions, enabling graphs of UE work with arbitrary data dependencies to execute deterministically within a single UE frame.Transactions are specified with begin_frame and end_frame contexts containing Python-implemented UE work graphs.
- Applications: SPEAR demonstrates applications spanning multiple embodied agents, city-scale rendering, procedural content generation, synchronized human-face views, MuJoCo co-simulation, and natural-language scene editing.The examples cover humans, cars, and robots across several in-the-wild UE projects and sample projects.
2 Related Work
Prior embodied-AI simulators include static-scan and fully interactive environments built with custom stacks that typically prioritize simulation speed over photorealism. UE’s Blueprints provide broad scripting access, but their unfamiliarity and proprietary binary format hinder text-based developer workflows.
- Simulators for Embodied AI: Custom embodied-AI simulators support navigation through static real-world scans and manipulation of fully interactive environments, typically prioritizing simulation speed over photorealism.
- Scripting Interfaces to the Unreal Engine: Blueprints expose nearly every class, function, and property visible to UE’s reflection system and can script standalone applications.
- Scripting Interfaces to the Unreal Engine: Blueprints are unfamiliar to most AI practitioners and use a proprietary binary format incompatible with text-based workflows such as diffing, merging, and AI coding assistance.
3 Programming Model
SPEAR’s programming model lets users express complex Unreal Engine work as Python-defined transactions while exposing reflection-visible UE functionality directly and extensibly. Transactions provide precise execution control within frames, with deterministic stepping achievable through pause–mutate–pause sequencing.
- Design Goals: The model prioritizes expressive, programmable, ergonomic, extensible, and composable control over Unreal Engine work without narrowing UE’s expressive power through mandatory domain abstractions.Optional domain-specific abstractions can be built on top of the model rather than required within it.
- Transactions: Users specify graphs of Unreal Engine work as transactions written in Python, gaining precise control over execution within and across individual UE frames.By default, each transaction executes within a single frame.
- Deterministic Execution: Deterministic stepping is achieved by starting UE paused and using each transaction to unpause the simulation, mutate game state, and pause it again.This approach keeps the UE application responsive to user input instead of blocking the game thread between transactions.
- Extensibility: All reflection-visible UE functionality is exposed as native Python functions and attributes, and new C++ functions or variables can be added through UFUNCTION or UPROPERTY annotations without modifying SPEAR.The annotations can be added in any C++ header, including headers outside the SPEAR codebase.
- Asynchronous Execution: Asynchronous operations prevent Python from outrunning the UE game thread by allowing at most one pending transaction at a time.A new begin_frame context blocks until work from the previous context finishes.
4 System Architecture
SPEAR uses a C++ client-server architecture with a nanobind Python wrapper, strongly typed entry points, and thread coordination that supports asynchronous, frame-synchronized UE work. Its interface maps structured NumPy data and UE objects across the boundary, with optional shared memory for efficiency.
- Client-server interface: SPEAR implements both client and server in C++ with rpclib, while nanobind provides the Python client wrapper and strongly typed server entry points.Server entry points accept standard containers and custom types, and clients call them like native C++ functions.
- Threading and frame execution: A dedicated server thread supports asynchronous operations while remaining synchronous with the Python thread, separating server-thread and game-thread execution.This arrangement simplifies the architecture while supporting the programming model’s asynchronous operations.
- Threading and frame execution: Thread-safe begin_frame and end_frame queues coordinate server requests that must access Unreal Engine’s game thread.The server places game-thread tasks into separate queues for begin_frame and end_frame work.
- Threading and frame execution: Python begin_frame and end_frame context managers send begin, execute, and end commands, while synchronous C++ entry points are invoked directly through nanobind.This lets Python call server functions such as load_class and spawn_actor without text-command parsing and imperative dispatch.
- Data exchange and memory: SpFunctions exchange named NumPy data arrays with shape and type metadata, named UE objects encoded as JSON strings, and miscellaneous user data strings.NumPy arrays are mapped to the named data-array representation at the client-server boundary and mapped back on output.
- Data exchange and memory: Optional interprocess shared memory lets Python allocate regions and create backed NumPy arrays for passing to SpFunctions without unnecessary copying.SPEAR exposes shared-memory allocation and deallocation through hand-crafted server entry points.
5 Results
SPEAR’s results demonstrate broad applicability and programmability through diverse applications that existing UE-based simulators cannot implement. The evaluation also compares its rendering performance with controlled plugin, standalone, and simulator baselines under specified hardware and scene-matching procedures.
- Example Applications: SPEAR supports a diverse collection of example applications, with additional applications provided in the supplementary material and public code repository.The examples are referenced in Figures 1, 6, 7, 8, and 9.
- Example Applications: Existing UE-based simulators cannot implement the demonstrated applications because they do not expose functionality needed to interact with UE’s procedural content system or control its path tracer.
- Rendering Performance: Rendering comparisons measure end-to-end delivery time from rendering to a user’s Python program and report frame time in milliseconds and frames per second.The evaluation includes plugin, standalone, and cross-simulator configurations at 1920×1080 resolution.
- Programmable Functionality: SPEAR exposes UE functionality through the reflection system, yielding more programmable functions and enabling a wider range of programs than existing simulators.The required functionality is exposed automatically through SPEAR’s architecture.
- Rendering Performance: The plugin comparison uses otherwise identical Unreal Engine projects and matched project settings to compare SPEAR with UnrealCV+ under controlled conditions.Both executables render the same view of the same scene.
- Rendering Performance: Cross-simulator comparisons use different scenes but match standalone rendering speeds as closely as possible because AirSim and CARLA environments are not directly comparable.SPEAR uses Epic Games’ HillsideSample, while AirSim and CARLA use their default environments; distant objects and rendering quality are adjusted for SPEAR.
6 Conclusions
SPEAR is presented as a Python-controlled Unreal Engine simulator with an expressive programming model and over 14K exposed UE functions. The authors envision it as a foundational data engine and bridge for computer vision, robotics, and embodied AI.
- Core contribution: SPEAR programmatically controls any Unreal Engine application through an expressive high-level Python programming model.Its architecture is designed for broad control of UE applications.
- Core contribution: 14K+ unique UE functions are exposed to Python, providing an order-of-magnitude increase in programmable functionality over existing UE-based simulators.The conclusion identifies programmability as a central advantage of SPEAR.
- Future impact: The authors propose SPEAR as a potential foundational data engine for computer vision, robotics, and embodied AI.This is presented as an anticipated role rather than an established result.
- Future impact: SPEAR could support agile-robot training in city-scale photorealistic environments and interactive world models of physical-world dynamics and spatial structure.These are described as near-future applications enabled by SPEAR’s capabilities.
- Future impact: Small Python programs provide programmability, while fast photorealistic rendering provides observability, positioning SPEAR as a bridge to internet-scale visual data.The supplied passage states that the conclusion continues beyond this point.