Source-linked AI summary
PIRM Challenge on Perceptual Image Enhancement on Smartphones: Report
Andrey Ignatov, Radu Timofte, Thang Van Vu, Tung Minh Luu, Trung X Pham, Cao Van Nguyen, Yongwoo Kim, Jae-Seok Choi, Munchurl Kim, Jie Huang, Jiewen Ran, Chen Xing, Xingguang Zhou, Pengfei Zhu, Mingrui Geng, Yawei Li, Eirikur Agustsson, Shuhang Gu, Luc Van Gool, Etienne de Stoutz, Nikolay Kobyshev, Kehui Nie, Yan Zhao, Gen Li, Tong Tong, Qinquan Gao, Liu Hanwen, Pablo Navarrete Michelini, Zhu Dan, Hu Fengshuo, Zheng Hui, Xiumei Wang, Lirui Deng, Rang Meng, Jinghui Qin, Yukai Shi, Wushao Wen, Liang Lin, Ruicheng Feng, Shixiang Wu, Chao Dong, Yu Qiao, Subeesh Vasu, Nimisha Thekke Madam, Praveen Kandula, A. N. Rajagopalan, Jie Liu, Cheolkon Jung
TL;DR
The paper reviews a smartphone-focused challenge for efficient perceptual image enhancement, addressing both conventional super-resolution and real-world photo enhancement. It describes the challenge’s resource-aware evaluation and reports that submitted solutions improved baseline efficiency and quality, defining the state of the art for smartphone image enhancement.
Problem
Image-enhancement challenges often prioritize conventional quality scores and computationally expensive models, motivating benchmarks for perceptual quality under smartphone deployment constraints.
Method
The challenge evaluates two tracks—4× bicubic super-resolution and iPhone 3GS-to-Canon 70D enhancement—using runtime, quantitative metrics, perceptual judgments, and smartphone deployment measurements.
Results
The proposed solutions significantly improved baseline results, and the challenge’s solutions defined the state of the art for image enhancement on smartphones.
Takeaways & Limitations
Efficient runtime and memory are treated as critical measures of successful image-enhancement deployment on real applications and mobile devices.
Abstract
from arXiv · showhide
This paper reviews the first challenge on efficient perceptual image enhancement with the focus on deploying deep learning models on smartphones. The challenge consisted of two tracks. In the first one, participants were solving the classical image super-resolution problem with a bicubic downscaling factor of 4. The second track was aimed at real-world photo enhancement, and the goal was to map low-quality photos from the iPhone 3GS device to the same photos captured with a DSLR camera. The target metric used in this challenge combined the runtime, PSNR scores and solutions' perceptual results measured in the user study. To ensure the efficiency of the submitted models, we additionally measured their runtime and memory requirements on Android smartphones. The proposed solutions significantly improved baseline results defining the state-of-the-art for image enhancement on smartphones.
1 Introduction
The challenge addresses whether perceptual image-enhancement quality can be achieved with smaller, resource-efficient networks suitable for smartphones. It benchmarks both perceptual results and deployment efficiency rather than optimizing conventional quality scores alone.
- Motivation: Existing image-restoration challenges primarily optimize quantitative metrics such as MSE, PSNR, SSIM, and MOS.Their prevailing recipe uses deeper architectures, more layers and filters, and longer training on many GPUs.
- Motivation: The paper questions whether marginal score improvements justify substantial increases in computational complexity.It considers whether smaller networks can deliver similar perceptual results on smartphones or tablets.
- Challenge contribution: PIRM 2018 benchmarks resource-efficient architectures for perceptual computer-vision tasks intended for mobile deployment.The challenge covers image super-resolution and image enhancement, while incorporating runtime and both quantitative and qualitative visual results.
- Challenge contribution: The challenge is presented as an initial benchmark for deploying deep-learning solutions that target high perceptual quality on mobile devices.Its scope connects perceptual performance with practical smartphone deployment constraints.
2 PIRM 2018 challenge
PIRM 2018 organizes a development, validation, and test process around two image-enhancement tracks. One reconstructs images from 4× bicubic downscaling, while the other maps iPhone 3GS photos to corresponding Canon 70D DSLR images.
- Challenge phases: The challenge proceeds through development, validation, and test phases.Participants receive data, validate solutions on a server leaderboard, and submit final models and results for testing.
- Tracks: PIRM 2018 consists of two tracks covering image super-resolution and image enhancement.The tracks address conventional reconstruction and real-world smartphone-photo improvement.
- Track A: Image super-resolution: The super-resolution track reconstructs an original image from a version downscaled bicubically by a factor of 4.Training uses 800 diverse high-resolution DIV2K images.
- Track B: Image enhancement: The enhancement track maps photos captured by an iPhone 3GS to corresponding photos captured by a Canon 70D DSLR.The task uses the DPED dataset, containing images captured simultaneously by smartphones and a high-end DSLR camera.
3 Scoring and validation
Submissions are evaluated with runtime, fidelity, and perceptual measures, supplemented by a large user study and several weighted score profiles. The scoring framework balances quantitative quality, visual quality, and speed across different validation priorities.
- Evaluation metrics: Submitted TensorFlow .pb models are evaluated using runtime, PSNR, and MS-SSIM metrics.Runtime is measured on 1280×720 images relative to the SRCNN baseline; PSNR measures fidelity and MS-SSIM measures perceptual score.
- User study: More than 2000 participants rated submitted visual results in a user study.For Track B, participants selected among four quality levels relative to the original input, and preferences were averaged into final MOS values.
- Scoring: The final submission score is calculated as a weighted sum of the component scores, including a runtime term.The runtime contribution uses γ · min(4, Time baseline / Time solution).
- Validation tracks: Three validation scores emphasize different trade-offs: score A favors quantitative results, score B favors perceptual results, and score C balances speed, visual quality, and quantitative quality.Their coefficient triplets differ across the two tasks, including score C values of (2, 200, 1.5) and (2, 200, 2.9).
4 Results
The challenge attracted over 100 validation submissions from more than 20 teams, with 12 teams reaching final testing. Across both tracks, solutions improved baseline efficiency and performance, while results also exposed limits in standard perceptual metrics.
- Challenge participation: More than 100 submissions came from over 20 teams during validation, while 12 teams entered the final test phase.Their models, code, and factsheets were evaluated in the final phase.
- Track A: Image super-resolution: Three to eight times faster than SRCNN, submitted super-resolution solutions also produced substantially better visual and quantitative results.Participants generally achieved this speedup relative to the SRCNN baseline.
- Track A: Image super-resolution: The super-resolution user study found nearly identical MOS scores across submissions except for the SRCNN baseline, limiting perceptual discrimination between methods.The submitted models mainly optimized Euclidean, MS-SSIM, and VGG-based losses rather than a strong adversarial loss.
- Track B: Image enhancement: The Mt.Phoenix enhancement solution achieved top MOS, best A, B, and C scores, and the fastest CPU and GPU runtime.On smartphones, it processed one HD-resolution photo in around 1.5 seconds on the Razer Phone and 2 seconds on the Huawei P20.
- Overall results: For both super-resolution and enhancement, diverse solutions surpassed the baselines and substantially improved efficiency in runtime and memory.The challenge’s solutions were reported as defining the state of the art for image enhancement on smartphones.
5 Proposed methods
The challenge submissions used diverse CNN designs for super-resolution and smartphone photo enhancement, emphasizing efficient architectures and task-specific training choices.
- TEAM ALEX: TEAM ALEX used a 20-residual-block network with reversible 4× desubpixel downsampling and two subpixel upsampling modules.The model was trained with combined MSE and VGG-based losses on task-specific patch sizes.
- KAIST-VICLAB: KAIST-VICLAB combined 4× downscaling and residual learning for track A, while its track B model used space-to-batch encoding and an adversarial discriminator.The track A network had eight convolutional layers and integrated pixel shuffle, error feedback, and xUnit components.
- Mt.Phoenix: Mt.Phoenix used a deep residual CNN with downsampling and deconvolution for super-resolution, and a U-net with global features and multiple losses for enhancement.The enhancement model included average-pooled bottleneck features and a global transform layer.
- CARN CVL: CARN CVL proposed an anchored regression network designed to trade off speed and accuracy through convolutional regression blocks.The challenge entry used five regression blocks, 16 anchors per block, and reduced feature layers.
- Other teams: Other teams proposed modified DPED, FCCN, Laplacian-pyramid, teacher-student, θ-inception, and multi-operating-point architectures for efficient enhancement.SNPR used three generator architectures with different operating points; G3 achieved the best speed-up ratio but lower performance than the DPED baseline.
PIRM 2018 Team
The paper is affiliated with the Computer Vision Lab at ETH Zurich, Switzerland.
- The paper lists the Computer Vision Lab, ETH Zurich, Switzerland as its affiliation.
TEAM ALEX
TEAM ALEX presented a fast image-enhancement approach centered on desubpixel down-sampling.
- TEAM ALEX’s work was titled “Fast and Efficient Image Quality Enhancement using Desubpixel Down-sampling.”
- The work was authored by Tung Luu, Trung Pham, and Cao Nguyen.
KAIST-VICLAB
KAIST-VICLAB contributed separate low-complexity approaches for perceptual super-resolution and detail enhancement with a relativistic discriminator.
- Track A: Track A was titled “A Low-Complexity Convolutional Neural Network for Perceptual Super-Resolution using Randomly-Selected Degraded LR and Enhanced HR.”
- Track B: Track B was titled “A Convolutional Neural Network for Detail Enhancement with the Relativistic Discriminator.”
- The listed authors were Jae-Seok Choi and Munchurl Kim.
Mt.Phoenix
This section identifies “Multi Level Super Resolution Net” and “Range Scaling Global U-Net for Perceptual Image Enhancement on Mobile Devices” as related work from Meitu Imaging & Vision Lab, China.
- “Multi Level Super Resolution Net ” is listed as related work.
- “Range Scaling Global U-Net for Perceptual Image Enhancement on Mobile Devices” is presented alongside the related work entry.
- The listed authors are Chen Xing, Xingguang Zhou, Jie Huang, Mingrui Geng, and Jiewen Ran.
CARN CVL
This section lists smartphone image enhancement and super-resolution works, including Convolutional Anchored Regression Network and several related approaches. It also identifies the authors and affiliations associated with these works.
- CARN CVL: The section names Convolutional Anchored Regression Network as one of the listed works.The listed authors are Eirikur Agustsson, Shuhang Gu, Radu Timofte, and Luc Van Gool, affiliated with ETH Zurich’s Computer Vision Lab.
- Related approaches: Several listed works address efficient, compact, fast, or perceptual image enhancement on smartphones.Examples include An Efficient and Compact Mobile Image Super-resolution with Fast Clique Convolutional Network, Perception-Preserving Convolutional Networks for Image Enhancement on Smartphones, and Fast Perceptual Image Enhancement.
- Related approaches: The section also lists methods for image super-resolution, image-to-image translation, DSLR-quality photo enhancement, and broader image enhancement.Named examples include Enhanced FSRCNN for Image Super-Resolution, Deep Networks for Image-to-image Translation with Mux and Demux Layers, Fast and Accurate DSLR-Quality Photo Enhancement, and Multiple Connected Residual Network for Image Enhancement on Smartphones.