Source-linked AI summary
SROBB: Targeted Perceptual Loss for Single Image Super-Resolution
Mohammad Saeed Rad, Behzad Bozorgtabar, Urs-Viktor Marti, Max Basler, Hazim Kemal Ekenel, Jean-Philippe Thiran
TL;DR
SISR perceptual losses can improve visual quality but remain limited when the same reconstruction objective is applied across semantically different regions. The paper introduces segmentation-derived OBB labels and targeted perceptual losses for boundaries and backgrounds, reporting more realistic textures, sharper edges, and stronger qualitative and user-study results than competing methods.
Problem
Whole-image perceptual loss ignores semantic information and therefore uses the same reconstruction treatment for different image regions.
Method
The method trains a CNN decoder with MSE, adversarial, and targeted perceptual losses guided by segmentation-derived Object, Background, and Boundary labels.
Results
The authors report more realistic textures, sharper edges, and superior qualitative and user-study results compared with state-of-the-art super-resolution methods.
Takeaways & Limitations
Region-specific perceptual objectives provide semantic control over super-resolution by applying edge-focused losses to boundaries and texture-focused losses to backgrounds.
Takeaways & Limitations
LPIPS can follow distortion-based metrics such as SSIM and does not necessarily imply photorealistic images.
Abstract
from arXiv · showhide
By benefiting from perceptual losses, recent studies have improved significantly the performance of the super-resolution task, where a high-resolution image is resolved from its low-resolution counterpart. Although such objective functions generate near-photorealistic results, their capability is limited, since they estimate the reconstruction error for an entire image in the same way, without considering any semantic information. In this paper, we propose a novel method to benefit from perceptual loss in a more objective way. We optimize a deep network-based decoder with a targeted objective function that penalizes images at different semantic levels using the corresponding terms. In particular, the proposed method leverages our proposed OBB (Object, Background and Boundary) labels, generated from segmentation labels, to estimate a suitable perceptual loss for boundaries, while considering texture similarity for backgrounds. We show that our proposed approach results in more realistic textures and sharper edges, and outperforms other state-of-the-art algorithms in terms of both qualitative results on standard benchmarks and results of extensive user studies.
1. Introduction
SISR reconstructs high-resolution images from low-resolution inputs, but whole-image perceptual loss ignores semantic differences between edges, textures, and other regions. The paper proposes targeted perceptual loss using segmentation-derived OBB labels to assign appropriate losses to boundaries and backgrounds.
- SISR recovers a high-resolution image from its low-resolution counterpart, while perceptual losses improve visual quality beyond pixel-wise reconstruction.
- Whole-image perceptual loss limits performance because it applies the same feature-level reconstruction treatment regardless of semantic region.
- Different CNN depths capture different information, with early features emphasizing edges and deeper features representing textures or semantics.
- The proposed method builds OBB labels from pixel-wise segmentation annotations and targets edge losses at boundaries and texture losses in background areas.
- The authors report more realistic textures, sharper edges, and improved qualitative and user-study results over other state-of-the-art algorithms.
2. Related work
Prior SISR methods improved reconstruction through deeper architectures, residual connections, perceptual objectives, and semantic priors. This paper focuses on exploiting semantic information within the objective function to produce photorealistic images with fine structures.
- Pixel-wise objectives commonly produce blurry, over-smoothed images because they average across plausible high-resolution solutions.
- Perceptual-loss methods improve visual quality by minimizing feature-space differences from representations extracted by pretrained networks such as VGG.
- Semantic-prior methods can recover textures faithful to categories, but some require segmentation maps at test time, while multitask learning jointly performs SR and segmentation.
- The paper investigates a novel way to exploit semantic information within images for photorealistic super-resolution with fine structures.
3. Methodology
The method trains a residual CNN decoder with MSE, adversarial, and targeted perceptual losses. OBB labels divide images into object, background, and boundary regions so different CNN feature levels can guide reconstruction of textures and edges.
- 3. Methodology: The decoder is trained with three objective terms: pixel-wise MSE, adversarial loss, and the proposed targeted perceptual loss.
- 3.1. Targeted perceptual loss: Targeted loss assigns mid-level features to background textures, early-layer features to object boundaries, and no perceptual loss to object regions.
- 3.1. Targeted perceptual loss: Background regions include sky, plant, ground, and water, whose texture similarity is measured using VGG-16 ReLU 4-3 features.
- 3.1. Targeted perceptual loss: Boundary regions are broadened into strips and compared using VGG-16 ReLU 2-2 features to emphasize low-level edges and blobs.
- 3.2. OBB: Object, background and boundary label: OBB labels are constructed from segmentation labels by detecting class edges, dilating them into boundary strips, and assigning object, background, or boundary classes.
- 3.3. Architecture: The decoder uses residual blocks and upsampling blocks, with the targeted loss trained alongside MSE and adversarial losses.
4. Experimental Results
The experimental section evaluates the proposed method through qualitative, quantitative, and running-cost analyses. The supplied passage introduces these evaluation dimensions without reporting their results.
- The experiments assess the method using qualitative results, quantitative metrics, and running-cost analysis after describing training parameters and datasets.
4.1. Dataset and parameters
The study trains on COCO-Stuff-derived OBB labels and uses a two-stage loss schedule combining pixel-wise, targeted perceptual, and adversarial losses.
- 4.1. Dataset and parameters: OBB labels are generated from 50K COCO-Stuff images by grouping Sky, Plant, Ground, and Water into Background classes.The images contain semantic labels for 91 segmentation classes.
- 4.1. Dataset and parameters: Training pretrains the decoder for 25 epochs with pixel-wise MSE, then adds targeted perceptual and adversarial losses for 55 epochs.The targeted-loss weights are α=2 × 10^-6 and β=1.5 × 10^-6.
- 4.1. Dataset and parameters: Figure 5 compares bicubic, SRCNN, SelfExSR, LapSRN, RCAN, SRGAN, SROBB, and HR images for baby and baboon examples.The examples come from Set5 and Set14.
4.2. Qualitative Results
SROBB produces sharper, more photo-realistic edges on benchmark examples and more natural textures on COCO-Stuff images than the compared methods.
- 4.2. Qualitative Results: SROBB produces more photo-realistic baboon reconstructions with sharper edges and remains competitive with SRGAN on the baby image.The comparison includes bicubic, SRCNN, SelfExSR, LapSRN, RCAN, and SRGAN on Set5 and Set14.
- 4.2. Qualitative Results: On COCO-Stuff, SROBB generates more realistic and natural textures, while ESRGAN can produce over-sharpened edges and unrealistic reconstructions.The compared methods were retrained or evaluated under the stated comparison setup.
4.3. Quantitative Results
Standard distortion and perceptual metrics do not reliably reflect visual quality, so the paper emphasizes user studies, where SROBB is strongly preferred and targeted loss wins its ablation.
- 4.3.1 SSIM, PSNR and LPIPS: LPIPS measures reference-based perceptual similarity using calibrated deep networks, but it can follow distortion metrics without implying photorealistic images.The paper notes a similar limitation for SSIM and PSNR.
- 4.3.1 SSIM, PSNR and LPIPS: Figure 6 presents zoomed COCO-Stuff crops comparing bicubic, RCAN, EnhanceNet, SRGAN, SFT-GAN, ESRGAN, SROBB, and HR outputs.Crops are enlarged by factors of 2 to 5.
- 4.3.1 SSIM, PSNR and LPIPS: SSIM, PSNR, and LPIPS comparisons on baby and baboon images do not reflect the superior reconstruction quality inferred from visual comparisons.The paper therefore uses the user study as its quantitative evaluation.
- 4.3.2 User study: SROBB receives 617 user votes, ahead of ESRGAN’s 436, SFT-GAN’s 223, SRGAN’s 201, and RCAN’s 33 votes.The study involved 46 participants comparing five reconstruction methods across outdoor COCO-Stuff images.
- 4.3.3 Ablation study: Targeted perceptual loss receives 1212 ablation-study votes versus 417 for standard perceptual loss and 49 for pixel-wise loss alone.Among 35 images, the targeted-loss objective wins by majority on 30 images.
4.4. Inference time
SROBB performs inference without semantic input and reaches 31.2 frames per second at 1024×768 output resolution on a GeForce GTX 1080 Ti.
- 4.4. Inference time: 31.2 frames per second is achieved at 1024×768 output resolution on one GeForce GTX 1080 Ti.Unlike other content-aware approaches, SROBB requires no semantic information at test time.
5. Conclusion
The paper introduces a targeted perceptual loss for CNN-based single-image super-resolution that assigns relevant loss terms to different semantic regions. OBB labels provide spatial control, and experiments show more pleasing results that outperform state-of-the-art methods.
- The targeted perceptual loss penalizes different image regions with relevant terms, such as edge loss for edges and texture loss for textures.
- OBB labels derived from pixel-wise segmentation labels provide spatial control over semantic information during training.
- Experimental results show that the proposed training produces more perceptually pleasing results and outperforms state-of-the-art super-resolution methods.