Source-linked AI summary
UnrealCV: Connecting Computer Vision to Unreal Engine
Weichao Qiu, Alan Yuille
TL;DR
Creating realistic, controllable virtual worlds with accessible ground truth is difficult, despite their value for computer vision. The paper introduces UnrealCV, an open-source UE4 plugin with server-client commands for accessing and modifying game worlds, and demonstrates synthetic dataset generation and deep-network testing. The paper also identifies limitations including restricted 3D-content variability, missing mesh structure, difficult physics simulation, and unresolved transfer from synthetic images.
Problem
Computer vision needs realistic virtual worlds with controllable agents and detailed ground truth, but creating such worlds and annotations is difficult.
Method
UnrealCV embeds a server plugin in UE4 games and provides an external client and commands to access internal data, control agents, modify worlds, and obtain ground truth.
Results
The authors demonstrate UnrealCV by generating a synthetic image dataset with rich ground truth and linking Caffe and Faster-RCNN to a virtual world for algorithm testing.
Takeaways & Limitations
UnrealCV provides an open-source bridge between UE4 virtual worlds and computer vision programs for creating, inspecting, and evaluating algorithms in realistic scenes.
Takeaways & Limitations
The authors note limited 3D-content variability, missing internal 3D-mesh structure, difficult realistic physics simulation, and unresolved transfer from synthetic images.
Abstract
from arXiv · showhide
Computer graphics can not only generate synthetic images and ground truth but it also offers the possibility of constructing virtual worlds in which: (i) an agent can perceive, navigate, and take actions guided by AI algorithms, (ii) properties of the worlds can be modified (e.g., material and reflectance), (iii) physical simulations can be performed, and (iv) algorithms can be learnt and evaluated. But creating realistic virtual worlds is not easy. The game industry, however, has spent a lot of effort creating 3D worlds, which a player can interact with. So researchers can build on these resources to create virtual worlds, provided we can access and modify the internal data structures of the games. To enable this we created an open-source plugin UnrealCV (http://unrealcv.github.io) for a popular game engine Unreal Engine 4 (UE4). We show two applications: (i) a proof of concept image dataset, and (ii) linking Caffe with the virtual world to test deep network algorithms.
1 Introduction
UnrealCV addresses the difficulty of creating realistic, controllable virtual worlds by adapting UE4 and its game-industry resources for computer vision. Its open-source plugin exposes internal world data so researchers can obtain ground truth and connect AI programs.
- Synthetic datasets provide large image collections with detailed annotation for training and testing computer vision models, but annotation is costly and time-consuming.
- Game-industry 3D worlds offer realistic resources for constructing virtual environments, while modifying individual proprietary games is impractical.
- UnrealCV is an open-source UE4 plugin that lets compiled games expose and modify internal data structures.
- External AI programs can use UnrealCV commands to obtain ground truth and control agents in UE4 virtual worlds.
- Unlike tools producing one virtual world or a synthetic dataset, UnrealCV is intended to help researchers create new virtual worlds.
2 Related Work
Prior work used virtual worlds, games, movies, and 3D repositories for robotics and computer vision, but these efforts generally emphasized physical simulation, single worlds, or datasets rather than a reusable UE4 extension.
- Robotics simulators commonly emphasize physical accuracy over visual realism, limiting their suitability for computer vision research.
- Computer vision researchers have built large 3D repositories and virtual scenes that can also be used with Unreal Engine and UnrealCV.
- Earlier projects converted movies, racing games, or Unity scenes into datasets or individual virtual worlds, whereas UnrealCV provides a tool for creating virtual worlds.
3 Unreal Engine
The paper selects UE4 as the platform because it is modifiable, visually realistic, well supported, and broadly used beyond games. Modifying the engine can affect games built on top of it.
- A game engine contains shared rendering and design components, so modifying it can affect all games built on top of it.
- UE4 provides access to realistic content from architectural visualization, urban scenes, outdoor environments, and digital humans.
- UE4 was chosen because it is open-source, produces realistic images, provides documented world-building tools, and supports VR and architectural visualization.
4 UnrealCV
UnrealCV extends UE4 through a server-client plugin architecture and commands that let external programs control virtual worlds, access annotations, and modify scene properties. Its modular, cross-platform design supports integration with research software.
- Using UE4 for virtual worlds requires programmable camera control, access to internal data for ground truth, and modification of properties such as lighting and materials.
- UnrealCV embeds a server plugin in compiled UE4 games and provides an external client for sending commands to the virtual world.
- The client-server system communicates through sockets and supports Python and MATLAB implementations, enabling clients across languages and platforms.
- UnrealCV commands use vget to retrieve scene information and vset to change world properties such as lighting, object position, and camera position.
- Hierarchical commands make the system extensible, while compiled worlds and multi-language support simplify use and integration with external programs.
5 Applications
The applications demonstrate UnrealCV’s use for generating varied synthetic image data with ground truth and for testing object detection under changing viewpoints.
- Synthetic image dataset: A UE4 technical-demo world provides an indoor scene with furniture and supports two demonstrations.The world is based on RealisticRendering2 and is downloadable from the project website.
- Synthetic image dataset: Random camera positions and two camera heights generate varied images, while lighting, materials, and object locations can also be changed.The camera heights correspond to human eye level and Roomba robot level.
- Synthetic image dataset: The dataset includes image-aligned ground truth such as depth, surface normals, and object instance masks.The generation procedure sets camera location and rotation before retrieving images and annotations.
- Diagnosing a deep network algorithm: A Faster-RCNN model trained on PASCAL is tested on sofa detection while camera viewpoints are varied and ground-truth boxes are derived from sofa instance masks.The virtual-world client controls the camera, and the detector is evaluated across views directed toward the sofa.
- Diagnosing a deep network algorithm: AP varies from 0.1 to 1.0 across sofa viewpoints, indicating strong viewpoint sensitivity.The authors suggest training biases may cause Faster-RCNN to favor specific viewpoints.
6 Conclusion
The conclusion presents UnrealCV as an open-source UE4 tool for constructing realistic virtual worlds and accessing their internal data for vision research. It also identifies limitations involving content variability, mesh structure, physics, and transfer from synthetic images.
- Contribution: UnrealCV plugs into UE4 to construct realistic virtual worlds from game, virtual reality, and architecture-visualization resources.The tool is open source and intended for researchers to use and extend.
- Capabilities: These virtual worlds provide access to internal data structures for extracting ground truth, controlling agents, and training and testing algorithms.The conclusion frames these capabilities as the tool’s research utility.
- Challenges: The authors identify limited 3D-content variability, missing internal mesh structure, difficult realistic physics simulation, and unresolved synthetic-to-real transfer as challenges.These are stated as ongoing challenges for computer vision use of virtual worlds.