PIRM Challenge on Perceptual Image Enhancement on Smartphones: Report

Introduction

• Current restoration challenges primarily reward quantitative scores, encouraging deeper architectures and longer training, while leaving open whether marginal gains justify the computational cost.The introduction raises the possibility that smaller, resource-efficient networks could achieve similar perceptual results on smartphones or tablets.Source: §1 · p. 1
• The mobile-device focus reflects the growing relevance of machine-learning and computer-vision problems on smartphones and tablets, including image enhancement and super-resolution [14].Source: §1 · p. 1
• PIRM 2018 benchmarks resource-efficient architectures for mobile deployment across image super-resolution and enhancement, using metrics that account for runtime and quantitative and qualitative results.The challenge is described as a first step toward benchmarking models aimed at high perceptual results on mobile devices.Source: §1 · p. 1

2 PIRM 2018 challenge

• Participants accessed data during development, tested solutions on a server leaderboard during validation, and submitted final results and models during testing.Source: §2 · p. 1

2.1 Track A: Image super-resolution

• Track A reconstructs images from bicubically downscaled inputs at factor 4, using DIV2K’s 800 high-resolution training images; sample SRGAN results appear in Figure 1.Source: §2.1 · p. 2, Figure 1 · p. 2

2.2 Track B: Image enhancement

• Track B maps photos from an iPhone 3GS to corresponding Canon 70D DSLR photos using the DPED dataset.DPED contains several thousand images captured simultaneously with three smartphones and one high-end DSLR camera.Source: §2.2 · p. 2

3 Scoring and validation

• Models submitted as TensorFlow .pb files were evaluated for HD-image speed against SRCNN, PSNR fidelity, and MS-SSIM perceptual quality.Speed was measured on 1280×720-pixel images.Source: §3 · p. 2
• More than 2,000 MTurk participants rated final-test results, and their Mean Opinion Scores replaced MS-SSIM in the scoring.For Track B, participants selected one of four quality levels relative to the original input; preferences were averaged across test images and then methods.Source: §3 · p. 2
• Three validation tracks emphasized quantitative results (A), perceptual results (B), or a balance of speed, visual quality, and quantitative scores (C).The score also included a weighted speed term, γ · min(4, Time baseline / Time solution).Source: §3 · p. 2

Image super-resolution:

• For the scoring scheme, the reported coefficients are A (4, 100, 1) and B (1, 400, 1), with no C coefficient included in the cited passage.Source: Image super-resolution · p. 2

Image enhancement:

• For image enhancement, (α, β, γ) coefficients were A (4, 100, 2), B (1, 400, 2), and C (2, 200, 2.9).Scoring scripts, pre-trained baseline models, and submission requirements were available in the challenge GitHub repository.Source: Image enhancement · p. 2

4.1 Image Super-Resolution

• Track A solutions were 3–8 times faster than SRCNN with better visual and quantitative results; TEAM ALEX led validation scores, while KAIST-VICLAB led runtime.KAIST-VICLAB processed HD images in under one second on both tested smartphones. MOS scores were nearly identical across submitted models, which largely lacked strong adversarial loss and instead optimized Euclidean, MS-SSIM, or VGG-based losses; only baseline SRCNN differed.Source: Table 1 · p. 3, §4.1 · p. 3

4.2 Image enhancement.

• Mt.Phoenix won Track B with top MOS and A, B, and C scores and the fastest CPU and GPU runtime, taking about 1.5 and 2 seconds per HD photo on the Razer Phone and Huawei P20.All submissions improved runtime and PSNR over SRCNN and DPED, but high PSNR did not ensure the best visual results, and MS-SSIM did not predict MOS well.Source: §4.2 · p. 3

4.3 Discussion

• The challenge treated runtime and memory as critical deployment measures; proposed solutions surpassed baselines across both tasks and defined the state of the art for smartphone image enhancement.Source: §4.3 · p. 3

5.1 TEAM ALEX

• TEAM ALEX used a 20-residual-block network that performed most computation after reversible 4× desubpixel downsampling, then restored resolution with subpixel modules.Training combined MSE and VGG-based losses; the desubpixel operation rearranged spatial features into channels without information loss.Source: §5.1 · p. 3, Figure 3 · p. 3

5.2 KAIST-VICLAB

• KAIST-VICLAB used an 8-layer residual super-resolution CNN with pixel shuffle, error feedback, and xUnit, and a separate encode-decoder design for photo enhancement.For super-resolution, training used 2,800 additional images and L1 loss; the enhancement model used space-to-batch downsampling, relativistic RGAN discrimination, and color and adversarial losses.Source: §5.2 · pp. 3–4

5.3 Mt.Phoenix

• Mt.Phoenix used a residual CNN with downsampling and deconvolution for super-resolution, and a U-net-style enhancement model augmented with global features and a global transform layer.The enhancement model combined L1, MS-SSIM, VGG, total-variation, and GAN losses.Source: §5.3 · p. 4, Figure 5 · p. 4

5.4 CARN CVL

• CARN CVL formulated super-resolution as anchor-based regression in convolutional blocks, with stacked blocks intended to improve performance while trading speed against accuracy.The challenge model used 5 regression blocks, 16 anchors per block, 2 feature layers, and 8 inner channels for 4× upscaling.Source: §5.4 · pp. 4–5

5.5 EdS; 5.6 IV SR+

• EdS modified the DPED ResNet with stride-2 convolutions and skip connections, while IV SR+ proposed FCCN with fast clique blocks built from bidirectional depthwise and pointwise convolutions.EdS trained for 33K iterations; IV SR+ used two feature-extraction convolutions, fast clique blocks, and two deconvolution layers.Source: §5.5 · p. 5, Figure 7 · p. 5, §5.6 · p. 5

5.7 BOE-SBG; 5.8 Rainbow

• BOE-SBG used a shared-parameter Laplacian-pyramid network for super-resolution and a separate enhancement network with information-preserving Mux/Demux layers; Rainbow used cascaded residual blocks for super-resolution and teacher-guided student training for enhancement.BOE-SBG combined channel-wise L1, contextual, VGG, color, total-variation, and adversarial losses for enhancement; Rainbow added feature-map distillation to its enhancement losses.Source: §5.7 · p. 5, Figure 9 · p. 5, Figure 10 · p. 5

5.9 MENet; 5.10 SuperSR; 5.12 Geometry

• MENet used parallel convolutional and deconvolutional paths for multiscale enhancement, SuperSR used space-to-depth downsampling and residual layers for super-resolution, and Geometry trained intermediate and final enhancement outputs with different losses.MENet added texture losses and fine-tuning; SuperSR restarted training with Charbonnier and MS-SSIM losses after L2 pretraining; Geometry used SSIM for its intermediate output and adversarial, smooth, and style losses for its final output.Source: §5.9 · p. 6, §5.10 · p. 6, §5.12 · p. 7

5.11 SNPR

• SNPR offered three enhancement generators with different operating points: G3 had the best speed-up ratio but lower performance than the DPED baseline, while G1 had the lowest speed-up ratio and comparable quality.The networks extract features at low image dimensions and concatenate them with full-resolution input features to compensate for spatial reduction.Source: §5.11 · p. 6, Figure 13 · p. 6

TEAM ALEX

• The listed work is “Fast and Efficient Image Quality Enhancement using Desubpixel Down-sampling”.Source: TEAM ALEX · p. 7
• Tung Luu, Trung Pham, and Cao Nguyen are named as the work’s authors.Source: TEAM ALEX · p. 7
• The listed affiliation begins with the Department of Electrical Engineering at KAIST.Source: TEAM ALEX · p. 7

• Title-A presents a low-complexity CNN for perceptual super-resolution using randomly selected degraded LR and enhanced HR.Source: KAIST-VICLAB · p. 7
• Title-B presents a CNN for detail enhancement with a relativistic discriminator.Source: KAIST-VICLAB · p. 7

Mt.Phoenix

• “Multi Level Super Resolution Net” is listed with bibliography marker.Source: Mt.Phoenix · p. 7
• “Range Scaling Global U-Net for Perceptual Image Enhancement on Mobile Devices” is the listed paper title.Source: Mt.Phoenix · p. 7
• The listed authors are Chen Xing, Xingguang Zhou, Jie Huang, Mingrui Geng, and Jiewen Ran, affiliated with Meitu Imaging & Vision Lab, China.Source: Mt.Phoenix · p. 7

CARN CVL

• The CARN CVL entry is titled “Convolutional Anchored Regression Network”.The listed authors are Eirikur Agustsson, Shuhang Gu, Radu Timofte, and Luc Van Gool, affiliated with the Computer Vision Lab at ETH Zurich, Switzerland.Source: CARN CVL · p. 7
• A separate listed work is titled “An Efficient and Compact Mobile Image Super-resolution with Fast Clique Convolutional Network.”Its listed authors are Yan Zhao, Gen Li, Tong Tong, and Qinquan Gao, with affiliations at Fuzhou University and Imperial Vision in China.Source: CARN CVL · p. 7

BOE-SBG

• Pablo Navarrete Michelini, Zhu Dan, and Hu Fengshuo are affiliated with BOE Technology Group Co., Ltd, China.Source: BOE-SBG · p. 8