PaperlayerView exact revision on arXiv

Source-linked AI summary

Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications

Matthias Müller, Vincent Casser, Jean Lahoud, Neil Smith, Bernard Ghanem

arXiv:1708.05869v2cs.CV

TL;DR

Computer vision research needs more varied annotated data and rigorous evaluation, especially for UAV tracking and realistic driving scenarios. Sim4CV addresses this with a photo-realistic, customizable Unreal Engine simulator integrating synthetic data, benchmark evaluation, and supervised DNN driving. The paper demonstrates the system on UAV tracking and autonomous driving, reporting broad functionality and performance beyond human drivers in its tested environments, while identifying sim-to-real transfer as an ongoing challenge.

  • Problem

    Annotated UAV datasets remain limited for rigorous airborne tracking evaluation, while existing driving simulators constrain scenario diversity and customization.

  • Method

    Sim4CV integrates photo-realistic Unreal Engine environments, physics-based vehicles and actors, automatic world generation, annotated synthetic data, tracking evaluation, and supervised DNN driving.

  • Results

    Sim4CV provides real-time synthetic benchmarks for UAV tracking and autonomous driving, and its driving approach outperforms human drivers across diversely arranged tested environments and maps.

  • Takeaways & Limitations

    The simulator offers a reusable, moderately integrated platform for evaluating vision problems and exploring environmental conditions and difficult tasks.

  • Takeaways & Limitations

    Transferring capabilities from simulated to real worlds remains unresolved because simulated and real-world appearance differences require reconciliation through transfer learning.

Abstract

from arXiv · show

We present a photo-realistic training and evaluation simulator (Sim4CV) with extensive applications across various fields of computer vision. Built on top of the Unreal Engine, the simulator integrates full featured physics based cars, unmanned aerial vehicles (UAVs), and animated human actors in diverse urban and suburban 3D environments. We demonstrate the versatility of the simulator with two case studies: autonomous UAV-based tracking of moving objects and autonomous driving using supervised learning. The simulator fully integrates both several state-of-the-art tracking algorithms with a benchmark evaluation tool and a deep neural network (DNN) architecture for training vehicles to drive autonomously. It generates synthetic photo-realistic datasets with automatic ground truth annotations to easily extend existing real-world datasets and provides extensive synthetic data variety through its ability to reconfigure synthetic worlds on the fly using an automatic world generation tool. The supplementary video can be viewed a https://youtu.be/SqAxzsQ7qUU

1 Introduction

Sim4CV is a customizable, physics-based Unreal Engine 4 simulator for generating annotated synthetic data, controlled experiments, and real-time evaluation across computer vision applications. It demonstrates these capabilities through UAV tracking and autonomous driving.

  • Simulator scope: Sim4CV combines car and UAV physics, animated agents, visual data capture, semantic outputs, tracking algorithms, evaluation tools, and TensorFlow-based learning.Its outputs include RGB images, depth, segmentation, and ground-truth labels.
  • Computer vision applications: Sim4CV supports synthetic-data generation and real-time evaluation for applications including object tracking, pose estimation, detection, and action recognition.Figure 1 distinguishes necessary from optional data types for each application.
  • World generation: Runtime procedural generation places configurable PBR assets and dynamic pedestrians or cars into diverse synthetic worlds.A GUI supports building-block placement and configuration for large-scale data generation and evaluation.
  • Case studies: The paper evaluates two use cases: real-time UAV tracking evaluation and autonomous car driving.The UAV case extends earlier work with a more extensive synthetic dataset.
  • Autonomous driving: The driving system trains a DNN with diverse synthetic visual data and automatically generated waypoint labels while running simulation and inference at real-time speeds.The approach is supervised and uses the simulator’s accompanying labels rather than manually collected training data.

2 Related Work

Prior simulators and datasets provide useful synthetic visual data but are often constrained by primitive rendering, fixed worlds, limited aerial coverage, or narrow driving environments. Sim4CV addresses these constraints through integrated tools, customizable world generation, and applications for UAV tracking and autonomous driving.

  • Learning from Simulation: Earlier physics-based simulators often lacked advanced shading, diverse assets, motion capture, and natural actor or vehicle animation.These limitations constrained visual realism and scenario variety.
  • Learning from Simulation: Photo-realistic game engines enabled synthetic datasets with pixel-accurate annotations, but several prior systems focused on data extraction rather than integrated experimentation.Virtual KITTI generated 35 video sequences and about 17,000 automatically annotated frames.
  • Learning from Simulation: Sim4CV integrates automatic world generation, external communication, and implemented UAV-tracking and autonomous-driving applications without extensive UE4 development or a host vision application.The package is designed for immediate use and straightforward integration of additional tools.
  • UAV Tracking: UAV tracking lacked sufficiently large and varied annotated benchmarks for rigorous precision and robustness evaluation in airborne scenarios.VIVID contained nine sequences with similar low-resolution content, sparse annotation, vehicle-only targets, and limited fixed-wing coverage.
  • UAV Tracking: Existing UAV methods commonly relied on feature-point or color-centric tracking, while Sim4CV enabled quantitative tracker evaluation across diverse photo-realistic scenarios.The simulator extends earlier work with a more extensive synthetic tracking dataset.
  • Autonomous Driving: Prior driving environments such as TORCS omitted urban, suburban, and rural complexities, while GTA V lacked interactive customization and repeatable control of views and agents.The cited limitations affected augmentation, evaluation, repeatability, and the variety available for DNN training.

3 Simulator Overview

Sim4CV is an installable Unreal Engine 4 simulator with configurable worlds, vehicles, external interfaces, and synthetic vision outputs. Its implementation supports broad application coverage while fully experimenting with object tracking and autonomous navigation.

  • Setup: Sim4CV is distributed as an installable video-game-like application with configurable settings, rich driving and flying worlds, multiple vehicles, and designed maps.An external map editor allows users to create their own maps.
  • Simulator capabilities: Unreal Engine 4 is repurposed for real-time RGB rendering and can output segmentation, bounding boxes, class labels, and depth.Multiple cameras can be configured within a scene.
  • Applications: Sim4CV generates synthetic data for twelve primary computer vision topics, including tracking, pose estimation, detection, navigation, reconstruction, and scene understanding.The paper fully implements and evaluates object tracking and autonomous navigation.
  • Applications: The simulator releases implementations for object tracking, autonomous navigation, and the general interface to third-party software.This release is intended to facilitate community use.
  • UE4 extensions: Sim4CV adds UE4-specific blueprints and C++ classes to support physics-based navigation, controllers, logging, replay, data augmentation, and outdoor world generation.Its interfaces include Python, C++, and Matlab socket communication.
  • Assets: The simulated asset collection combines purchased UE4 marketplace assets with an author-designed UAV model based on a real-world challenge vehicle.The vehicle variety includes cars, an RC truck, and UAVs.

4 Tracking

Sim4CV provides a physics-based UAV tracking test bed with synthetic imagery, automatic ground truth, and controlled online and offline evaluation. The experiments compare trackers across diverse speeds and maps, revealing substantial trade-offs between accuracy, robustness, and processing speed.

  • Overview: Sim4CV evaluates state-of-the-art trackers on automatically annotated synthetic sequences and integrates them with the simulator for online UAV control.The tracking case study includes both offline benchmarking and direct in-the-loop evaluation under close to real-world conditions.
  • UAV Physics Simulation and Control: The UAV simulation models flight physics, camera imagery, depth masks, target state, and vehicle state while keeping the UAV position unknown to controllers and trackers.A gimbaled camera and physics-based flight controller support tracking experiments, while each frame logs bounding box, position, orientation, and velocity data.
  • UAV Physics Simulation and Control: Automatic world generation enables controlled tracking experiments by varying environments and isolating attributes while providing precise position, orientation, and velocity comparisons.The simulator also measures dynamic frame-rate effects, target–UAV trajectory error, ground-truth-controller differences, and long-term tracking.
  • Offline Evaluation: MEEM achieves the best offline precision and success while exceeding 30fps, whereas C-COT has comparable performance below 1fps and MOSSECA exceeds 200fps.The evaluation used low-resolution imagery on a powerful workstation, so the reported speeds are tied to that setting.
  • Online Evaluation: MOSSECA is the only tracker to complete map2 at medium speed and reaches about 75% completion at high speed, outperforming the other trackers.MEEM completes about 75% of map2 at medium speed but fails early at high speed; SRDCF and SAMF fail earlier under harder conditions.
  • Qualitative Results: C-COT loses the target quickly after initialization because its runtime is below 1fps, while SAMF fails early during fast turning and MOSSECA drifts after temporary target disappearance.Under heavy tree occlusion, all evaluated trackers are affected, with SAMF having already lost the target in the described sequence.

5.1 Overview

The autonomous-driving case study separates pathway estimation from vehicle control, using a waypoint-predicting DNN and an algorithmic controller evaluated in simulation.

  • The DNN predicts waypoints ahead of the car, while an algorithmic controller converts them into steering and throttle commands.
  • The waypoint approach uses autogenerated labels and is designed to generalize across cars without retraining the network.
  • Synthetic environments support automatic training-data generation and online evaluation through real-time communication between Sim4CV and TensorFlow.

5.2 Data Acquisition

Sim4CV generates diverse virtual driving environments and automatically labels roadway geometry for waypoint-learning data, using configurable maps and surroundings.

  • An overhead-view editor builds environments from neighborhoods to entire cities using standardized blocks or randomly generated road networks.
  • Generated environments are loaded directly into Sim4CV for training and testing, with urban scenes constructed from the editor.
  • The simulator moves the car along the road system at fixed distances and renders images without requiring human steering or accelerator input.
  • Each non-augmented view is paired with 4 waypoints spaced 2 meters apart, covering a prediction range of 2 to 8 meters.
  • Waypoints are encoded relative to the car by horizontal and vertical offsets projected onto the viewing axis.
  • Training uses 16 maps with a 12-4 training-validation split and 4 additional test maps, each represented in desert and urban surroundings.

5.3 DNN-Training

The waypoint network is selected for real-time performance and predicts waypoint offsets from RGB images, with an optional goal input for guided driving.

  • The architecture was chosen through extensive experiments while optimizing for a small network with high real-time performance.
  • An optional goal value bypasses the convolutional layers to encode the desired direction at intersections for guided driving.
  • Training uses L2 loss, Adam optimization with a base learning rate of 5e−5, and early stopping when validation error stops decreasing.
  • The best-performing network uses a 7-layer architecture to predict two offsets for each waypoint from a 320 × 180 RGB image.

5.4 Vehicle Controller and Scoring

The controller derives steering and throttle from predicted waypoint offsets, and scoring measures how closely the simulated car follows the ideal pathway.

  • Steering is computed from the first waypoint’s horizontal and vertical offsets using θ = arctan(h / v).
  • Throttle is based on the absolute horizontal offset of the last waypoint, decreasing as the car approaches a turn.
  • The system produces a smooth trajectory without explicitly enforcing temporal smoothness during training or evaluation.
  • Testing records car position and averages Euclidean deviation from the ideal pathway across all tracks.
  • The critical region is bounded by lane geometry, with d = 1 m placing the car exactly at either edge of the right lane.
  • Cumulative histograms report the percentage of time spent within each deviation range using 5 cm bins.

5.5 Evaluation

Evaluation shows that random viewpoint augmentation produces the strongest driving performance, while the system also supports guided driving, lane changes, and obstacle avoidance. The network stays closer to the ideal pathway than human drivers and avoids critical lane-edge regions.

  • Augmentation: Without viewpoint augmentation, the driving system fails on every test track because it does not learn to recover from lane drift.Adding only a few additional views per original frame is sufficient for good performance.
  • Comparison to human performance: The random-view model stays within [−25 cm, 25 cm] for over 95% of driving time and almost never leaves [−50 cm, 50 cm].The fixed-view model reaches 89% and 97% for the corresponding ranges.
  • Comparison to human performance: Average deviation is 7.12 cm for the random-view model and 14.61 cm for the fixed-view model, versus 30.17 cm for the best human test subject.Both models outperform the reported human subject, while the random-view model is more accurate.
  • Augmentation: Randomly augmented training outperforms fixed-offset training, even when the fixed-view model uses 100% more synthetic views and 50% more total data.The best fixed configuration uses only yaw offsets in [−30°, 30°].
  • Comparison to human performance: The network-driven approach reaches 100% cumulative performance at about 60 cm and never navigates close to lane edges, outperforming the human drivers.The comparison uses cumulative histograms of deviation from the ideal pathway.
  • Additional driving capabilities: Guided driving encodes left, straight, and right goals as −1, 0, and 1 to produce corresponding waypoint predictions at intersections.The approach also extends to lane changing and visually modeled obstacle avoidance.

5.6 Discussion

The discussion presents a modular waypoint-estimation approach that makes guided driving, lane changing, and obstacle avoidance straightforward. It reports strong performance across varied environments without requiring manually collected training data.

  • Modular driving approach: Decoupling deep-network pathway estimation from vehicle controls makes lane changes, obstacle avoidance, and guided driving straightforward to implement.Vehicle or behavior changes can be applied on the controller side without changing the learned network.
  • Training data: The approach works without human-generated or hand-crafted training data, although manually collected data can be included when available.Training uses synthetic data and accompanying labels in the described system.
  • Evaluation: Across diversely arranged environments and maps, the approach outperforms the capabilities of human drivers by far.The paper evaluates effectiveness by measuring driving performance in different environments and maps.

6 Conclusions and Future Work

Sim4CV is presented as a high-fidelity, end-to-end simulator and evaluation suite for computer-vision applications, demonstrated through tracking and autonomous driving. Future work targets richer scenarios, reinforcement learning, and transfer from simulation to the real world.

  • Conclusions: Sim4CV combines real-time synthetic benchmarking and evaluation for vision-based tracking and autonomous driving with moderate integration effort.The simulator supports exploration of environmental conditions and difficult vision tasks that are hard to control or replicate in the real world.
  • Future work: Future driving scenarios can include stop lights, pedestrians, and AI-controlled cars, but these elements were not explored in this paper.The authors plan to use them for more sophisticated driving models.
  • Future work: Future work plans to pursue reinforcement learning and use depth measurements such as stereoscopy, RGB-D, or LiDAR point clouds as auxiliary inputs.These measurements are proposed for rewards and penalties in reinforcement-learning training.
  • Future work: Transfer of capabilities from simulated to real worlds remains future work requiring greater environmental diversity and methods such as deep transfer learning.The authors identify appearance differences between simulated and real environments as a transition challenge.
  • Broader impact: The generic, open-source simulator and its deep-learning interface are expected to support development and evaluation of new models and tracking algorithms.The authors also anticipate applications in other autonomous-navigation and AI tasks.
Loading 1708.05869v2…