Source-linked AI summary
High-Resolution Photorealistic Image Translation in Real-Time: A Laplacian Pyramid Translation Network
Jie Liang, Hui Zeng, Lei Zhang
TL;DR
High-resolution photorealistic image translation is constrained by the cost of processing full-resolution feature maps and by the need to preserve content details. LPTN translates attributes on reduced-resolution low-frequency components and progressively refines high-frequency details; the authors report real-time 4K translation with comparable or superior performance. The method cannot generate novel content details and may introduce halo artifacts in day-to-night translation.
Problem
High-resolution photorealistic I2IT must balance efficient inference with avoiding content distortions, while existing methods can incur heavy costs from high-resolution feature-map convolutions.
Source: §1 · pp. 1–2Method
LPTN uses a lightweight network to translate attributes on reduced-resolution low-frequency components and progressive masking to refine high-frequency details.
Source: §1 · p. 2, §3.1 · p. 4Results
The authors report real-time 4K photorealistic I2IT with comparable or superior transformation performance against existing methods.
Source: §1 · p. 2Takeaways & Limitations
The reported approach reduces computation on high-resolution images while retaining transformation performance and image detail reconstruction.
Source: §5 · p. 8LPTN cannot generate novel content details, and progressive masking may introduce halo artifacts in day-to-night translation.
Source: §4.3 · p. 7
Abstract
from arXiv · showhide
Existing image-to-image translation (I2IT) methods are either constrained to low-resolution images or long inference time due to their heavy computational burden on the convolution of high-resolution feature maps. In this paper, we focus on speeding-up the high-resolution photorealistic I2IT tasks based on closed-form Laplacian pyramid decomposition and reconstruction. Specifically, we reveal that the attribute transformations, such as illumination and color manipulation, relate more to the low-frequency component, while the content details can be adaptively refined on high-frequency components. We consequently propose a Laplacian Pyramid Translation Network (LPTN) to simultaneously perform these two tasks, where we design a lightweight network for translating the low-frequency component with reduced resolution and a progressive masking strategy to efficiently refine the high-frequency ones. Our model avoids most of the heavy computation consumed by processing high-resolution feature maps and faithfully preserves the image details. Extensive experimental results on various tasks demonstrate that the proposed method can translate 4K images in real-time using one normal GPU while achieving comparable transformation performance against existing methods. Datasets and codes are available: https://github.com/csjliang/LPTN.
1. Introduction
Photorealistic image translation must handle high-resolution images efficiently while avoiding content distortions. LPTN assigns attribute translation to low-frequency components and adaptively refines high-frequency details, enabling real-time 4K translation with comparable or superior performance.
1. Introduction
- High-resolution I2IT is difficult because deep processing of full-resolution feature maps is costly, while shallow models may fail to reconstruct complex details faithfully.Encoding-decoding approaches are often limited to low resolution or slow inference; down- and up-sampling can still require expensive high-resolution convolutions.Source: §1 · pp. 1–2
- Day-to-night image differences are concentrated in low frequencies, while high-frequency components retain more similar textures that can be adaptively refined.The reported high-frequency MSEs are about 1/71 and 1/65 of the low-frequency MSE; the pyramid bands also have tapering resolutions.Source: §1 · p. 2
- LPTN translates illumination and color attributes on low-frequency components and adaptively refines high-frequency content details with progressive masks.A lightweight network translates the low-frequency component, and a tiny network predicts a mask on the smallest high-frequency component for progressive upsampling.Source: §1 · p. 2
- LPTN enables real-time photorealistic I2IT on 4K images while achieving comparable or superior performance against state-of-the-art methods.The authors report favorable qualitative and quantitative results on photorealistic I2IT applications.Source: §1 · p. 2
2. Related Work
Prior photorealistic translation methods face efficiency and reconstruction trade-offs at high resolution. The paper motivates Laplacian pyramids as a reversible way to separate frequency bands and translate attributes without heavy high-resolution convolutions.
2.1. Photorealistic Image Translation
- Photorealistic translation approaches based on encoding and decoding, disentanglement, or style transfer can remain computationally costly at high resolution or introduce structural distortions.Some style-transfer methods also require a reference image for each input, while the proposed LPTN uses the training-data distribution at test time.Source: §2.1 · pp. 2–3
2.2. Laplacian Pyramid
- A Laplacian pyramid linearly decomposes an image into high- and low-frequency bands that permit exact reconstruction from the recorded residuals.Its low-pass prediction uses weighted neighboring pixels, and the decomposition iterates on the reduced-resolution image.Source: §2.2 · p. 3
- LPTN uses low-frequency translation for illumination and color changes and lightweight high-frequency refinement to avoid costly convolutions on large components.The authors connect this frequency-specific strategy to real-time implementation on 4K images.Source: §2.2 · p. 3
3. Laplacian Pyramid Translation Network
LPTN decomposes an image into band-pass and low-frequency components, translates global attributes on the reduced-resolution residual, and progressively refines high-frequency details. It is trained end-to-end with reconstruction and adversarial losses.
3.1. Framework Overview
- The framework decomposes an image into band-pass components H and a low-frequency residual I_L, whose resolutions taper across the pyramid levels.The decomposition is invertible, and the blurred low-frequency residual reflects global image attributes in a content-independent manner.Source: §3.1 · pp. 3–4
3.2. Translation on Low-Frequency Component
- LPTN translates illumination and color attributes on the reduced-resolution low-frequency component, using a lightweight network with five residual blocks.The paper contrasts fixed-kernel pyramid decomposition with autoencoders that use heavy convolutions over the whole image.Source: §3.2 · p. 4
3.3. Refinement of High-Frequency Components
- A mask learned from the low-frequency components and the coarsest high-frequency component refines that high-frequency component, then is progressively upsampled to refine finer-scale components.Bilinear upsampling and lightweight convolutions exploit the similarity of high-frequency bands to save computation compared with direct convolution on large components.Source: §3.3 · pp. 4–5
3.4. Learning criteria
- Unsupervised training combines an image-space reconstruction loss with an adversarial loss computed using a multi-scale discriminator.The total loss is L = L_recons + λL_adv, with λ balancing the two losses.Source: §3.4 · p. 5
4. Experiment
Experiments evaluate day-to-night and summer-to-winter translation, FiveK photo retouching, module ablations, visual quality, runtime, and user preferences. LPTN is reported to preserve image details while extending photorealistic translation to 4K.
4.1. Setup
- The experiments use Flickr day/night and summer/winter datasets from 1080p to 4K, plus FiveK expert-C retouching targets with 4,500 training pairs and 500 evaluation pairs.Quantitative metrics use paired FiveK samples at test time.Source: §4.1 · p. 5
4.2. Ablation Study
- Removing high-frequency refinement preserves attribute translation but blurs textures (PSNR 20.87); removing low-frequency instance normalization causes excessive translation and oversharpened details.The PSNR is averaged over 500 test images under the specified setting.Source: §4.2 · p. 5, Figure 3 · p. 6
4.3. Visual Comparisons
- Across day→night, summer→winter, and retouching, LPTN compares favorably with CycleGAN, UNIT, and MUNIT on photorealism and translation, while extending to higher resolutions such as 4K.The visual comparison with those methods is conducted at 1080p because of their memory limitations.Source: §4.3 · p. 6, Figure 4 · p. 7
- LPTN's high-frequency masking may introduce halo artifacts in day-to-night translation, and the Laplacian-pyramid approach cannot generate novel content such as Cityscapes images from semantic labels.The authors suggest sparse convolution on high-frequency components as a possible solution to the halo issue.Source: §4.3 · p. 7
4.4. Quantitative Examinations
- About ×80 speed-up over CycleGAN on 1080p at L=4, and real-time 4K inference at L=5; LPTN performs favorably against general I2IT and photo-retouching methods.The runtime comparison averages 50 tests; Table 1 reports FiveK retouching comparisons.Source: §4.4 · pp. 7–8, Table 1 · p. 7, Table 2 · p. 8
4.5. User Study
- In day→night user preferences, LPTN scores 78.3% for photorealism and 50.2% for transformation effectiveness; summer→winter results are described as similar.Twenty participants judged results for each translation task.Source: §4.5 · p. 8, Table 3 · p. 8
5. Conclusion
LPTN uses an invertible Laplacian pyramid to separate visual attributes from textures, translating and refining them at lower resolutions. Across photorealistic I2IT tasks, it runs faster while achieving comparable or superior translation performance, including real-time translation of 4K images on a desktop GPU.
5. Conclusion
- Across photorealistic I2IT tasks, LPTN runs much faster while achieving comparable or superior translation performance, including real-time processing of 4K images on a desktop GPU.The framework reduces computational burden on high-resolution images while keeping transformation performance.Source: §5 · p. 8
- LPTN uses an invertible Laplacian pyramid to disentangle domain-specific visual attributes and textures across tapering resolutions, learning translation and refinement networks on low-resolution components.This reduces computational burden when handling high-resolution images while keeping transformation performance.Source: §5 · p. 8
- A progressive masking strategy adaptively refines high-frequency components to generate photorealistic results.Source: §5 · p. 8