HyPER-GAN: Hybrid Patch-Based Image-to-Image Translation for Real-Time Photorealism Enhancement in Game Engines

Stefanos Pasios; Nikos Nikolaidis

Source-linked AI summary

HyPER-GAN: Hybrid Patch-Based Image-to-Image Translation for Real-Time Photorealism Enhancement in Game Engines

Stefanos Pasios, Nikos Nikolaidis

arXiv:2603.10604v3Mar 11, 2026cs.CV

TL;DR

Rendered images still differ from real-world imagery, while existing enhancement models can alter scene content and demand substantial computation. HyPER-GAN combines lightweight image translation with hybrid patch training, achieving 6x higher 1080p FPS while improving photorealism and preserving semantic consistency.

Problem
Rendered images retain a sim-to-real appearance gap that can limit the real-world generalization of vision models trained exclusively on game-engine data.
Method
HyPER-GAN uses a lightweight U-Net-style image translator trained with paired rendered images and unpaired matched real-world patches.
Results
6x higher 1080p FPS (12.61 to 78.90 FPS) accompanies improved photorealism and semantic consistency across within- and cross-engine evaluations.
Takeaways & Limitations
HyPER-GAN supports real-time photorealism enhancement while maintaining temporal consistency and stronger content preservation across evaluated environments.
Takeaways & Limitations
Further optimization of real-time performance and visual realism is needed while maintaining comparable or better content preservation.

Abstract

from arXiv · show

Generative models are increasingly used in video game engines to enhance the photorealism of rendered images for visual synthetic data generation and simulation applications. However, they often introduce artifacts that alter the content of the original rendered scenes and require high computational resources, which limit their utilization for the photorealism enhancement of training and evaluation data, as well as their integration in the rendering pipelines of game engines. In this paper, we propose Hybrid Patch Enhanced Realism Generative Adversarial Network (HyPER-GAN), a hybrid image-to-image translation framework that is based on a lightweight U-Net-style generator capable of performing real-time inference. The framework is trained using paired rendered and photorealism-enhanced images, complemented by a novel hybrid training strategy that incorporates matched patches from unpaired real-world images to improve content preservation and further enhance the visual realism that can be achieved by the lightweight generator. Experimental results demonstrate that HyPER-GAN achieves a 6x increase in frames per second at 1080p in comparison with state-of-the-art lightweight paired image-to-image translation methods, while also increasing, in both within- and cross-engine evaluations, the photorealism of the rendered images without significantly compromising semantic consistency. Moreover, it is illustrated that HyPER-GAN maintains temporal consistency and that the proposed hybrid training strategy improves content preservation and visual realism in within-engine and increases the robustness in cross-engine evaluations compared to training the framework solely with paired rendered and photorealism-enhanced images. Code and pretrained models are publicly available at: https://github.com/stefanos50/HyPER-GAN

1 Introduction

Video game engines support computer-vision research but retain a sim2real appearance gap, motivating HyPER-GAN, a lightweight paired image-to-image framework with hybrid patch training for real-time photorealism enhancement and content preservation.

Motivation: Game-engine renders remain significantly different from real-world images despite advances such as Unreal Engine 5’s Lumen, creating a simulation-to-reality appearance gap.These engines are used to generate scenarios that are expensive, unsafe, or impractical to simulate in the real world.
Prior approaches: Image-to-image translation reduces the sim2real gap while being more computationally efficient and less prone to hallucinations than diffusion models.Methods are categorized as paired or unpaired according to whether pixel-aligned source–target image pairs are available during training.
Challenge: Paired photorealism enhancement can produce images that deviate from rendered content, threatening semantic consistency required by tasks such as semantic segmentation and object detection.Maintaining consistency with ground-truth annotations is especially important for computer-vision algorithms.
Proposed framework: HyPER-GAN combines a lightweight U-Net–style generator with paired supervision and matched patches from an unpaired real-world dataset.The design targets real-time inference on high-resolution rendered images and easy integration into rendering pipelines.
Contributions: 6x improvement in FPS is reported for HyPER-GAN, alongside enhanced photorealism and better semantic consistency in within- and cross-engine experiments.The hybrid strategy also improves content preservation and visual realism within-engine, while increasing robustness in cross-engine evaluations versus paired-only training.

2 Related Work

Prior work uses appearance-translation and generative models to reduce the sim2real appearance gap in game-engine imagery. Methods evolved from unpaired approaches using auxiliary G-Buffers toward lightweight frameworks intended for simulator integration and higher efficiency.

Appearance translation: Game-engine images, including GTA-V renders, motivate appearance-translation techniques and generative models for reducing the sim2real appearance gap.These methods adapt rendered images toward the visual characteristics of real-world images.
Unpaired methods: EPE uses additional game-engine G-Buffers, such as depth and albedo, to improve realism, semantic consistency, and temporal stability.EPE is an unpaired image-to-image translation method.
Unpaired methods: Approximately 2 FPS at 957 × 526 on an RTX 3090 is reported for EPE.
Simulator integration: CARLA2Real adapts a lightweight EPE variant for CARLA and aligns rendered images with Cityscapes and Mapillary Vistas in an unpaired translation setting.It achieves slightly above 2 FPS at 960×540 on an RTX 4090 without further optimization such as TensorRT.
Lightweight paired methods: REGEN uses EPE outputs to train a more lightweight paired image-to-image translation approach.It is described as a two-stage framework combining a robust unpaired method with a lightweight paired method.

3 HyPER-GAN

HyPER-GAN combines paired rendered/photorealism-enhanced data with indexed real-world patches and a lightweight U-Net/ PatchGAN architecture. During inference, it operates as a standalone feed-forward generator without auxiliary inputs.

Datasets and preprocessing: HyPER-GAN uses rendered images, photorealism-enhanced paired targets, and real-world images, with all datasets processed through the same preprocessing pipeline.The paired targets are generated by a robust unpaired image-to-image translation model using real-world images as targets.
Real-world dataset indexing: Real-world images are indexed by extracting four 196×196 non-overlapping patches and storing VGG-16 feature embeddings in FAISS using L2 distance.Images are resized to 512 × 512 before patch extraction, and embeddings come from the third convolutional layer of VGG-16 block 4.
Generator and Discriminator: The generator is a lightweight U-Net with three downsampling and three upsampling stages, increasing channels from 3 to 256, while the discriminator is PatchGAN-style.The generator also includes four residual bottleneck blocks, and the discriminator evaluates local texture realism through patch-level processing.
Similarity Search: For each generated image, four patches are matched to spatially corresponding target patches and nearest real patches, forming generated and real batches of eight patches.Nearest-neighbor matching uses the FAISS-indexed real-world database in VGG-16 feature space to reduce target artifacts and align semantically similar content.
Loss Functions: HyPER-GAN trains its generator with LSGAN adversarial and reconstruction losses, using λ = 10 to weight the L1 distance.The discriminator assigns 1 to real patches and 0 to generated patches, while the generator seeks generated patches classified as real.
Photorealism enhancement: At inference, HyPER-GAN discards FAISS and the discriminator and directly maps a rendered image to a photorealism-enhanced output without additional inputs.This standalone feed-forward design avoids requirements such as G-Buffers or semantic segmentation annotations.

4 Experiments and Results

Experiments evaluate HyPER-GAN’s real-time performance, photorealism, semantic consistency, cross-engine robustness, and temporal consistency across rendered and real-world datasets. HyPER-GAN delivers real-time inference while generally improving realism and content preservation over lightweight baselines and its paired-only variant.

Real-Time Benchmarking Results: HyPER-GAN reaches 30 FPS at 1440p and near-real-time performance of 15 FPS at 4K on an RTX 4090.Real-time benchmarking reports inference latency, FPS, and VRAM utilization over 100 images across 720p, 1080p, 1440p, and 4K.
Within-engine Results: In within-engine evaluation, HyPER-GAN achieves the best or second-best score across all metrics and improves CMMD over rendered PFD images from 5.274 to 3.080 for CS and 4.296 to 2.827 for MV.It also improves visual realism and content preservation over FastCUT and semantic consistency over REGEN.
Within-engine Results: HyPER-GAN avoids artifacts observed in EPE outputs, including hallucinated vegetation, geometric water distortions, and unrealistic vehicle glossiness.The comparison attributes this advantage particularly to fewer hallucinations.
Cross-engine Results: Across VKITTI2 and OffRoadSynth, HyPER-GAN achieves the best or second-best performance across most metrics while cross-engine content-preservation failures intensify in the off-road environment.The cross-engine evaluations test environments that differ substantially from the urban PFD training environment.
Temporal Consistency: On CrowdFlow, HyPER-GAN’s temporal-consistency results are comparable with REGEN, slightly improving the background while slightly increasing crowd error.Background improvement is attributed to smoothing small rendered-image artifacts, whereas increased crowd error can result from shadows occluding individuals.

5 Conclusions

HyPER-GAN combines a lightweight U-Net–style generator with hybrid training using matched real-world patches to enhance rendered-image photorealism in real time while preserving content. Experiments show stronger semantic robustness, improved realism–consistency balance, reduced artifacts in unseen environments, and temporal consistency, while future work targets further optimization.

Contributions: HyPER-GAN combines a lightweight U-Net–style generator with matched real-world patches to improve content preservation and visual realism within real-time game-engine constraints.The hybrid strategy augments paired training with matched patches from unpaired real-world images.
Experimental findings: HyPER-GAN outperforms existing lightweight paired image-to-image methods in semantic robustness and balances semantic consistency with visual realism in cross-engine evaluations.These properties support photorealism enhancement of rendered images for computer-vision research.
Experimental findings: HyPER-GAN-EO improves semantic consistency and visual realism within-dataset and is less prone to visual artifacts in unseen environments, while HyPER-GAN maintains temporal consistency.The hybrid training approach is responsible for the HyPER-GAN-EO improvements described in the conclusion.
Future work: Future research should further optimize real-time performance and improve visual realism while maintaining or exceeding current content preservation.The paper identifies CMMD as a realism measure and LPIPS and mIoU as content-preservation measures.

A Real-Time Benchmarking inside UE5

Within UE5, the GPU must both render engine images and run the enhancement model. At 1080p, UE5+HyPER-GAN sustains near-real-time performance, whereas UE5+REGEN falls substantially below 30 FPS.

HyPER-GAN performance: 24–28 FPS: UE5+HyPER-GAN maintains an acceptable frame rate close to the 30 FPS real-time target.This result is reported across the simple and more complex UE5 environments.
REGEN comparison: UE5+REGEN produces FPS that deviate significantly from real-time performance at 30 FPS.The comparison evaluates REGEN and HyPER-GAN while the UE5 GPU simultaneously renders and performs inference.
Benchmark setup: The UE5 benchmark measures average FPS with each model integrated into the engine, covering simple and complex environments at 1080p.Testing used an Intel i7 14700F CPU, NVIDIA RTX 4090 GPU with 24GB VRAM, and 32GB DDR4 memory.

B Additional Qualitative Examples

Figure 9 provides additional qualitative examples of HyPER-GAN translating PFD (GTA-V) test frames toward real-world CS and MV datasets. The examples show updates to mountain vegetation and road textures.

Additional Qualitative Examples: Figure 9 shows HyPER-GAN translation results on PFD (GTA-V) test frames toward the real-world CS and MV datasets.The examples illustrate changes to rendered-scene elements, including mountain vegetation and road textures.