PaperlayerView exact revision on arXiv

Source-linked AI summary

A Hybrid Approach for Closing the Sim2real Appearance Gap in Game Engine Synthetic Datasets

Stefanos Pasios

arXiv:2605.02291v1cs.CV

TL;DR

The paper addresses the persistent sim2real appearance gap that limits the use of game-engine synthetic datasets in real-world computer vision. It compares FLUX.2-4B Klein with REGEN and proposes a hybrid pipeline combining diffusion-based enhancement with Im2Im distribution matching. REGEN outperforms FLUX.2-4B Klein, while their combination achieves better visual realism than either model alone and maintains semantic consistency.

  • Problem

    A persistent visual gap between game-engine synthetic images and real-world images limits the utilization of synthetic datasets in real-world applications.

  • Method

    The paper compares FLUX.2-4B Klein and REGEN and combines them by applying diffusion enhancement followed by Im2Im real-world distribution matching.

  • Results

    REGEN outperforms FLUX.2-4B Klein, while the hybrid approach achieves better visual realism than either model individually and remains faithful to synthetic ground-truth annotations.

  • Takeaways & Limitations

    Matching the real-world dataset distribution is more important for closing the appearance gap, while combining it with diffusion changes improves visual realism and preserves semantic consistency.

  • Takeaways & Limitations

    Temporal inconsistencies limit the approach to frame-level synthetic-data tasks, and its diffusion component prevents real-time application.

Abstract

from arXiv · show

Video game engines have been an important source for generating large volumes of visual synthetic datasets for training and evaluating computer vision algorithms that are to be deployed in the real world. While the visual fidelity of modern game engines has been significantly improved with technologies such as ray-tracing, a notable sim2real appearance gap between the synthetic and the real-world images still remains, which limits the utilization of synthetic datasets in real-world applications. In this letter, we investigate the ability of a state-of-the-art image generation and editing diffusion model (FLUX.2-4B Klein) to enhance the photorealism of synthetic datasets and compare its performance against a traditional image-to-image translation model (REGEN). Furthermore, we propose a hybrid approach that combines the strong geometry and material transformations of diffusion-based methods with the distribution-matching capabilities of image-to-image translation techniques. Through experiments, it is demonstrated that REGEN outperforms FLUX.2-4B Klein and that by combining both FLUX.2-4B Klein and REGEN models, better visual realism can be achieved compared to using each model individually, while maintaining semantic consistency. The code is available at: https://github.com/stefanos50/Hybrid-Sim2Real

I. INTRODUCTION

Synthetic game-engine datasets offer controllable, automatically annotated imagery, but a persistent sim2real appearance gap limits their real-world use. The paper proposes combining diffusion-based enhancement with image-to-image distribution matching to improve realism while preserving annotation fidelity.

  • Motivation: Game engines generate large-scale synthetic datasets with accurate annotations in controllable and customizable environments.This is useful when real-world data collection is time-consuming, costly, or unsafe.
  • Motivation: A persistent sim2real appearance gap remains despite advances in modern game-engine rendering technologies.The gap is described as a visual difference between synthetic and real-world images.
  • Existing approaches: Image-to-image translation matches real-world dataset distributions, whereas diffusion methods can make stronger geometry and material changes.The two method families therefore offer complementary capabilities for photorealism enhancement.
  • Proposed approach: The proposed approach combines FLUX.2-4B Klein with REGEN to improve photorealism while retaining semantic consistency.The approach uses diffusion enhancement followed by image-to-image distribution matching.

II. PHOTOREALISM ENHANCER

The proposed hybrid photorealism-enhancement approach is organized as a two-phase pipeline. It first enhances photorealism with diffusion and then matches the resulting image to a real-world dataset distribution using Im2Im translation.

  • Pipeline: The hybrid approach contains two phases: diffusion-based photorealism enhancement and Im2Im real-world dataset distribution matching.These phases define the overall structure of the proposed pipeline.
  • Diffusion phase: The first phase applies diffusion-based processing to enhance the photorealism of a synthetic image.
  • Distribution-matching phase: The second phase performs Im2Im translation toward the distribution of a real-world dataset.

A. Diffusion-based Photorealism Enhancement

The diffusion phase uses FLUX.2-4B Klein to transform a game-engine image into a more photorealistic counterpart. Its lightweight hardware requirements and RGB-only input make it applicable to existing synthetic datasets without additional control signals.

  • Diffusion enhancement: FLUX.2-4B Klein processes a game-engine synthetic image to produce a photorealism-enhanced counterpart.
  • Model selection: FLUX.2-4B Klein requires roughly 13GB of VRAM and can run on consumer-grade hardware such as an NVIDIA RTX 3090.
  • Input requirements: FLUX.2-4B Klein requires only an RGB image and does not require additional control signals such as semantic segmentation maps.This supports use with pre-existing synthetic datasets that were not exported with such information.

B. Im2Im Real-World Dataset Distribution Matching

The Im2Im phase feeds the diffusion-enhanced image into REGEN to add the distribution and characteristics of a selected real-world dataset. This further reduces the appearance gap while avoiding additional inputs such as depth.

  • Pipeline input: The diffusion-enhanced image is passed to an Im2Im translation model in the second phase.
  • Distribution matching: The trained Im2Im model adapts the diffusion output toward the distribution and characteristics of a target real-world dataset.It adds the complexities and characteristics associated with that specific dataset.
  • REGEN: REGEN is selected because it removes the requirement for additional inputs such as depth while performing real-world distribution matching.

A. Synthetic Datasets and Metrics

The experiments use synthetic datasets from two game engines and evaluate photorealism against real-world references while measuring semantic and structural preservation.

  • Synthetic Datasets: VKITTI2 contains 2,126 Unity-generated dash-cam images with semantic segmentation maps and camera intrinsics, while the GTA-V dataset contains 456 UAV-perspective images with bounding-box annotations.VKITTI2 covers 15 object categories; the GTA-V dataset covers 5.
  • Metrics: CMMD measures similarity between real-world reference datasets and synthetic or enhanced datasets, with lower values indicating greater visual similarity.The metric was selected because user studies found it aligned with human perception and judgment.
  • Metrics: Semantic preservation is assessed by comparing model performance on original and enhanced images using segmentation and object-detection metrics.The supplied passage introduces this evaluation alongside CMMD but truncates the metric names and details.

B. Experimental Setup

The experiments compare FLUX, REGEN, and their hybrid on synthetic datasets, including sequential enhancement and evaluation with pretrained computer-vision models.

  • Models: FLUX.2-4B and REGEN are pretrained models that had not seen VKITTI2 or GTA-V during training.REGEN was trained on synthetic CARLA images, while FLUX.2-4B is a zero-shot image-generation diffusion model.
  • Visual Comparison: Visual examples compare the VKITTI2 input with FLUX, REGEN, and FLUX+REGEN outputs for KITTI and CS variations.The figure presents the methods as alternative or combined photorealism-enhancement outputs.
  • Processing Pipeline: For VKITTI2, FLUX first enhances photorealism, after which REGEN is applied to produce KITTI- and CS-oriented variants, including the FLUX+REGEN hybrid.The passage defines the individual REGEN and hybrid processing variants but truncates the final pipeline description.
  • Evaluation: Semantic preservation is evaluated on enhanced images using pretrained Mask2Former for VKITTI2 segmentation and YOLO26m for GTA-V object detection.For GTA-V, mAP@50 is calculated on the synthetic images and enhanced variations; similar values indicate semantic consistency.

C. Results and Discussion

REGEN generally matches real-world dataset distributions better than FLUX, while the hybrid combines distribution matching with stronger geometry and material changes and preserves semantics.

  • Results and Discussion: REGEN produces a more significant CMMD reduction than FLUX in most cases, indicating stronger similarity to the target real-world distributions.Lower CMMD indicates higher similarity; the comparison covers VKITTI2 and GTA-V against KITTI and CS references.
  • Results and Discussion: FLUX introduces significant geometry and material changes, whereas REGEN transforms images toward the target real-world distributions and characteristics.The visual comparison includes KITTI and CS dataset variations.
  • Results and Discussion: The FLUX+REGEN combination produces photorealistic images containing both improved geometry and materials and real-world distributional characteristics.The hybrid is described as combining the complementary aspects observed in the individual methods.
  • Semantic Consistency: Mask2Former mIoU is matched or increased on enhanced VKITTI2 images, with the highest value for the CS-oriented FLUX+REGEN variant.YOLO26m mAP@50 remains similar between GTA-V synthetic images and the KITTI and CS FLUX+REGEN variants.

D. Limitations

The approach is constrained by temporal inconsistencies in diffusion-based enhancement and by the inability to operate in real time.

  • Limitations: Temporal inconsistencies in diffusion-based methods limit the approach to frame-level synthetic-dataset tasks rather than sequential visual data such as videos.The paper lists image classification, object detection, semantic segmentation, and depth estimation as applicable frame-level tasks.
  • Limitations: Because the approach relies on diffusion, it cannot be applied in real time, limiting its use in simulations.The paper notes that DLSS 5.0 combined with REGEN could potentially address these limitations.

IV. CONCLUSIONS

The study concludes that REGEN outperforms FLUX.2-4B Klein because matching the real-world dataset distribution is more important for reducing the sim2real appearance gap. Combining both models improves visual realism while preserving semantic consistency.

  • Matching the real-world dataset distribution is more important for closing the sim2real appearance gap than diffusion-based enhancement alone.
  • REGEN outperforms FLUX.2-4B Klein in photorealism enhancement of synthetic datasets.
  • The hybrid FLUX.2-4B Klein and REGEN approach achieves better visual realism than either model individually.
  • The hybrid photorealism-enhanced images remain semantically consistent with the synthetic data annotations, evaluated using mIoU and mAP@50.
Loading 2605.02291v1…