Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach

2. Related Work

• Pixel-level losses preserve fidelity but can produce over-smoothed details, whereas perceptual and adversarial losses improve realism but adversarial training can introduce artifacts.The related-work discussion also describes efforts to reduce GAN-generated artifacts and the challenge of real-world degradations.Source: §1 · pp. 1–2, §2 · p. 2

1. Introduction

• Existing SD-based SR methods can entangle pixel fidelity and semantic enhancement, while sequential restoration-then-enhancement methods depend on the earlier restoration's accuracy.Multi-step guidance methods may also lack precise control and efficiency, while users differ in their fidelity-versus-detail preferences.Source: §1 · p. 2
• PiSA-SR separates pixel-level and semantic-level enhancement into two LoRA spaces on pre-trained SD, with separate losses and guidance scales for adjustment without retraining.The pixel-level LoRA uses ℓ2 loss; the semantic-level LoRA uses LPIPS and classifier score distillation losses.Source: §1 · p. 2
• PiSA-SR reports superior SR performance to existing SD-based models and adjustable SR styles; increasing pixel guidance removes degradations but can over-smooth, while high semantic guidance can add artifacts.The two guidance scales let users trade pixel fidelity against semantic detail during inference.Source: §1 · p. 2

2. Related Work

• Diffusion-based SR methods use pre-trained diffusion priors, but prior approaches still face pixel-versus-semantic trade-offs and often lack flexible style adjustment.The related work describes methods including restoration before diffusion enhancement, degradation-aware prompting, and one-step distillation.Source: §2 · pp. 2–3

3.1. Model Formulation

• The model learns the residual between LQ and HQ latents, focusing on high-frequency information in the HQ latent rather than less relevant LQ information.A scale factor λ controls the residual added to the LQ latent; it is fixed at 1 during training.Source: §3.1 · pp. 3–4

3.2. Dual-LoRA Training

• Dual-LoRA training first optimizes a pixel-level LoRA, then freezes it while training a semantic-level LoRA, keeping the two objectives distinct.Only the semantic-level LoRA is updated in the second stage, reducing interference from pixel-level degradations.Source: §3.2 · p. 4

3.3. Pixel-level and Semantic-level Losses

• The pixel-level ℓ2 loss removes degradation and enhances edges but yields smooth outputs, while LPIPS and classifier score distillation (CSD) provide semantic enhancement.CSD contributes more to semantic enhancement than the zero-guidance VSD component and avoids VSD's bi-level optimization, reducing memory use and improving training stability.Source: §3.3 · pp. 4–5

3.4. The Inference Process of PiSA-SR

• At inference, the merged PiSA-LoRA enables one-step reconstruction, while λpix and λsem adjust the pixel-level and semantic-level contributions to suit user preferences.The difference between outputs with both LoRAs and with only the pixel-level LoRA is used to separate semantic details from pixel-level degradations.Source: Figure 3 · p. 5, §3.4 · p. 6

4.1. Experimental Settings

• Training uses SD 2.1-base for ×4 SR, with LSDIR and the first 10K FFHQ images, RealESRGAN-generated pairs, and 4K and 8.5K iterations for pixel- and semantic-level LoRAs.Evaluation uses synthetic DIV2K crops and real-world RealSR and DrealSR data; metrics include fidelity, perceptual, distributional, and no-reference measures.Source: §4.1 · p. 6

4.2. Experiments on Adjustable SR

• Increasing λpix continuously improves CLIPIQA and MUSIQ, while PSNR peaks at λpix = 0.5 and LPIPS peaks at λpix = 0.8 (0.2612).Increasing λsem also improves CLIPIQA and MUSIQ, with a higher upper bound, while PSNR decreases and LPIPS peaks at λsem = 0.8 (0.2465).Source: §4.2 · pp. 6–7, Table 1 · p. 6
• Increasing λpix removes degradation and sharpens output, but excessive pixel guidance loses details; increasing λsem enriches details, but λsem=1.5 introduces unnatural artifacts.The visual comparisons show users can favor fidelity or semantic enhancement through scale selection.Source: §4.2 · p. 7

4.3. Comparisons with State-of-the-Arts

• PiSA-SR uses one inference step and outperforms other methods on LPIPS and DISTS, as well as CLIPIQA, MUSIQ, and MANIQA.The comparison covers synthetic and real-world test datasets; PiSA-SR’s default model has the fewest parameters and fastest inference time among SD-based SR methods.Source: §4.3 · pp. 7–8, Table 2 · p. 7, Table 3 · p. 8
• Visual comparisons report realistic, richer details from PiSA-SR, while SeeSR can produce over-enhanced or over-smoothed details and OSEDiff’s restored details can be limited.PiSA-SR’s restored structures align with the input LQ image.Source: §4.3 · p. 7, Figure 5 · p. 8
• PiSA-SR’s adjustable setting requires two diffusion steps and slightly more inference time than its default setting.The additional time allows users to tailor SR outputs to their preferences; the default setting does not require a semantic extractor during inference.Source: §4.3 · p. 8, Table 3 · p. 8

Conclusion

• PiSA-SR disentangles pixel-level and semantic-level objectives into two LoRA modules, using ℓ2 for pixel-level optimization and LPIPS and CSD for semantic-level optimization.Adjustable guidance scales let users customize SR results during inference without retraining.Source: §5 · p. 8

Limitation

• The adjustable version incurs a slight inference-time increase over the default version.The pixel-level LoRA handles varied degradations, which may be insufficient for heavy degradations.Source: §5 · p. 8
• Future work will explore separate LoRA spaces for different degradations and image-adaptive scale factors.Source: §5 · p. 8

Dual-LoRA Approach”

• The supplementary file lists derivations, GAN-based comparisons, a user study, additional visual examples, and ablation studies.Source: Dual-LoRA Approach” · p. 12

• The derivation identifies H as the estimated HQ latent, c as a prompt extracted from zsem, and ϵreal as the pre-trained SD output with classifier-free guidance.It also defines zt as the noisy semantic output and wt as a timestep-dependent scalar weight introduced in DMD.Source: §6 · p. 12
• The distribution matching gradient is calculated within the latent space rather than the noise domain.Source: §6 · p. 12
• Applying classifier-free guidance and merging timestep-related weights rewrites Eq. (4) into the form of Eq. (5).Source: §6 · p. 12

• PiSA-SR achieves the best NIQE, CLIPIQA, MUSIQ, and MANIQA performance across three benchmark datasets among the compared GAN-based methods.The comparison includes RealESRGAN, BSRGAN [65], and LDL [29].Source: §7 · p. 12
• PiSA-SR demonstrates competitive results on LPIPS and DISTS, indicating a balance between perceptual quality and content fidelity.Source: §7 · p. 12
• Visual comparisons report more realistic rope and leaf textures in PiSA-SR outputs than in the GAN-based methods.The cited examples are regular rope textures and intricate leaf textures.Source: §7 · p. 12

• With pixel-level guidance fixed at 1.0, participants compared outputs generated using semantic guidance factors of 0.6 and 1.2.Source: §8 · p. 15
• The study treated selection of the 1.2 semantic-guidance output as evidence of improved semantic quality from increasing that guidance factor.Source: §8 · p. 15
• Across 500 votes from 10 participants on 50 different low-quality images, the approach had a 98% positive selection rate, supporting its effectiveness.Source: §8 · p. 15

• Increasing pixel-level enhancement gradually reduces degradation and sharpens edges.Source: §9 · p. 15
• Excessive pixel-level enhancement can blur finer details, such as a lady’s hair accessory.Source: §9 · p. 15
• Increasing semantic-level enhancement improves scene details but may introduce artifacts when over-enhanced.Examples include seagull wings, sea ripples, and island trees.Source: §9 · p. 15

10. More visual comparisons with DM-based SR methods

• PiSA-SR reconstructs more accurate structures and more realistic details than other DM-based SR methods in the visual comparisons.Examples include the red flag’s structure, the flower’s texture, and water ripples.Source: §10 · p. 15

11.1. Effectiveness of the dual-LoRA training

• Dual-LoRA training balances fidelity and perceptual quality, unlike pixel-only training with poor no-reference metrics or semantic-only training with reduced reference scores.The reported comparison is on the RealSR test dataset.Source: §11.1 · p. 15, Table 5 · p. 17

11.2. Comparison between CFG and the proposed semantic-level guidance

• Changing λ_sem from 1.0 to 1.2 raises MUSIQ from 70.15 to 71.60, while higher CFG scales can degrade reference- and no-reference-based metrics.Increasing CFG scale in one-step DMs does not improve no-reference metrics.Source: §11.2 · p. 17, Table 8 · p. 17

11.3. Impact of LoRA rank

• Increasing semantic-level LoRA rank improves semantic metrics but lowers fidelity scores, while increasing pixel-level rank improves fidelity but reduces some image-detail metrics.The authors choose rank 4 for both LoRA modules as a balance.Source: §11.3 · p. 17, Table 6 · p. 17

11.4. Impact of the input timestep

• Increasing the input timestep enhances semantic details but worsens fidelity; PiSA-SR uses timestep 1 to retain LQ image information.The authors attribute the trade-off to stronger denoising at larger timesteps, which improves generation but reduces preservation of input structures.Source: §11.4 · pp. 17–18, Table 7 · p. 17

11.5. Comparison with OSEDiff

• CSD avoids bi-level optimization, reducing training memory and iteration time, while residual learning accelerates convergence versus OSEDiff.The reported convergence comparison shows PiSA-SR restoring clearer text during the first 2000 training iterations.Source: §11.5 · p. 18, Figure 9 · p. 18, Figure 10 · p. 18
• PiSA-SR consistently demonstrates superior LPIPS and MANIQA performance to OSEDiff throughout training, although OSEDiff initially leads in MANIQA.The comparison measures reference-based LPIPS and no-reference-based MANIQA over training iterations.Source: Figure 9 · p. 18, §11.5 · p. 19