PaperlayerView exact revision on arXiv

Source-linked AI summary

Coloring the Noise: Adversarial Sobolev Alignment for Faithful Image Super Resolution

Hongbo Wang, Huaibo Huang, Pin Wang, Jinhua Hao, Chao Zhou, Ran He

arXiv:2605.23264v2cs.CVcs.AI

TL;DR

Faithful image super-resolution is limited by a spectral mismatch between isotropic generative priors and natural image statistics. ASASR aligns optimization with natural spectral decay using Sobolev geometry and a Riesz-grounded adversary, achieving superior spectral consistency and structural fidelity across evaluations.

  • Problem

    Isotropic generative priors remain spectrally misaligned with natural image statistics, limiting faithful image super-resolution.

  • Method

    ASASR uses Sobolev Spectral Rectification to color transport noise and a Riesz-grounded adversary to target structural deviations.

  • Results

    ASASR achieves superior performance over leading generative methods, particularly in preserving spectral consistency and fine-grained structural fidelity.

  • Takeaways & Limitations

    ASASR provides a spectral-geometric framework for balancing perceptual realism with structurally faithful restoration across diverse degradations and backbones.

  • Takeaways & Limitations

    The parametric adversary assumes sufficient representational capacity to approximate the Sobolev-optimal direction in function space.

Abstract

from arXiv · show

Generative priors in Image Super-Resolution (SR) often compromise faithful restoration, we attribute this limitation to a fundamental spectral misalignment between isotropic objectives and the intrinsic natural image manifold. While Direct Preference Optimization offers a path to alignment, its reliance on spectrally flat Gaussian noise fails to distinguish authentic high-frequency details from hallucinations. To bridge this geometric gap, we propose ASASR, a theoretically grounded framework that recasts the generative flow into a Sobolev-induced Riemannian geometry by explicitly coloring the noise transition kernel to mirror natural spectral decay. Driving this geometric alignment, we integrate a parametric adversary grounded in the Riesz Representation Theorem, which synthesizes targeted negative samples equivalent to worst-case Sobolev gradients to direct optimization along the tangent space of plausible structural failures. Extensive evaluations demonstrate that ASASR outperforms leading generative baselines, particularly in preserving spectral consistency and structural fidelity, offering a robust solution that effectively mitigates artifacts.

1. Introduction

The paper attributes unfaithful super-resolution to spectral misalignment between isotropic generative priors and natural image statistics. ASASR addresses this through Sobolev Spectral Rectification and Adversarial Manifold Guidance, improving spectral consistency and structural fidelity.

  • Motivation: Generative priors synthesize realistic textures, but supervised training limits faithful restoration by anchoring optimization to synthetic degradation priors rather than authentic natural-image statistics.
  • Motivation: Standard DPO has limited efficacy in SR because its isotropic Gaussian parameterization uses a spectrally flat prior that diverges from intrinsic spectral decay.
  • Method: Sobolev Spectral Rectification colors transport noise with a structured covariance matrix that mirrors natural-texture spectral density and reshapes optimization toward manifold adherence.
  • Method: Adversarial Manifold Guidance synthesizes targeted negative samples equivalent to worst-case Sobolev gradients, directing optimization along the tangent space of plausible structural failures.
  • Experiments: Extensive evaluations against leading diffusion-based and GAN-based approaches show that ASASR balances fidelity and realism while preserving spectral consistency and fine-grained structural fidelity.

2. Background

The background establishes ASASR on conditional Flow Matching, trajectory-wise Direct Preference Optimization, and Sobolev spaces that encode natural spectral regularity. Together, these foundations motivate replacing isotropic pixel-space geometry with frequency-aware alignment.

  • Flow Matching: Conditional Flow Matching interpolates between Gaussian prior noise and high-resolution data conditioned on the low-resolution input.The framework uses x1 ∼ q(x1|c) and x0 ∼ N(0, I) connected by a conditional probability path.
  • Flow Matching: A velocity network learns the conditional vector field governing the interpolated flow.The target field is ut(x|x0, x1) = x1 − x0.
  • Direct Preference Optimization: Direct Preference Optimization aligns the generative prior with human perception using preference triplets and deterministic ODE trajectories.The objective is extended to trajectory space to avoid intractable likelihood computation.
  • Sobolev Space: Sobolev space Hs(Ω) quantifies image regularity by incorporating smoothness constraints and frequency-dependent penalties on Fourier coefficients.Unlike L2, the Sobolev formulation enforces a natural spectral distribution with energy concentrated toward lower frequencies.

3. Manifold Rectification in Sobolev Space

The section identifies isotropic ℓ2 optimization as imposing uniform spectral weighting and causing high-frequency misalignment. It rectifies this geometry with colored Gaussian noise, whose precision operator induces a weighted Sobolev manifold that emphasizes fine-grained errors and improves spectral fidelity.

  • Euclidean Spectral Misalignment: Isotropic Gaussian transitions induce Euclidean geometry, reducing the likelihood-ratio objective to differences of squared ℓ2 residual norms.The residual γθ measures the difference between model prediction and the target vector field.
  • Euclidean Spectral Misalignment: The ℓ2 formulation uniformly weights all frequencies, producing spectral indifference that fails to counteract inherent spectral bias.Parseval’s theorem exposes the uniform frequency weighting underlying the spatial error objective.
  • Sobolev Spectral Rectification: Sobolev Spectral Rectification replaces identity covariance with structured spectral covariance Σs, changing optimization from isotropic noise to colored Gaussian noise.The resulting likelihood is governed by a Mahalanobis distance shaped by the precision matrix Σs^-1.
  • Sobolev Spectral Rectification: Because Σs^-1 amplifies high-frequency components, the objective penalizes fine-grained discrepancies more strongly and lifts optimization onto the weighted Sobolev manifold Hs(Ω).This operator recovers the Sobolev inner product and reformulates the likelihood ratio using squared Sobolev norms.
  • Spectral Fidelity: The Sobolev constraint closely aligns with the empirical natural-image spectral distribution, preserving fine-grained structural fidelity unlike the ℓ2 baseline.The ℓ2 baseline exhibits noticeable high-frequency decay, while the Sobolev formulation drives preference optimization through frequency-weighted restoration energy.

4. Adversarial Manifold Guidance

Adversarial Manifold Guidance (AMG) synthesizes spatially aligned, realistic hard negatives by modeling reconstruction artifacts and sharing noise across winner and degraded trajectories. It targets spectrally coherent hallucinations through Sobolev-guided perturbations that expose misaligned model confidence.

  • Motivation: Standard super-resolution datasets provide regression pairs rather than comparative triplets, while text-to-image preference data lacks spatial correspondence, preventing aligned negative construction.
  • Adversarial Manifold Guidance: AMG trains a parametric adversary on baseline outputs to mimic realistic reconstruction failures and synthesize targeted negatives such as text deformations and architectural distortions.
  • Aligned Sampling: Sharing the same noise realization across winner and degraded trajectories isolates content-dependent perceptual degradation and produces a counterfactual negative with precise semantic alignment.
  • Hard-Negative Construction: AMG targets Misaligned Confidence, where generative models assign high likelihood to structurally degraded samples containing coherent spectral artifacts such as textural distortions or aliasing.
  • Hard-Negative Construction: The spectral preconditioner Σs prevents collapse into trivial random noise and steers perturbations toward visually degraded states where models remain deceptively confident.
  • Theoretical Guarantee: Under sufficient representational capacity, minimizing the adversary’s energy subject to ∥Aϕ∥Hs ≤ εt recovers the Sobolev-optimal direction through spectral duality.

5. Experiments

Experiments evaluate ASASR across synthetic and real-world super-resolution benchmarks, perceptual and structural metrics, user preferences, downstream vision tasks, and component ablations. Across these settings, ASASR demonstrates strong perceptual, spectral, structural, semantic, and fidelity performance, with Sobolev guidance and s = 1.5 identified as effective design choices.

  • Experimental Setup: Training uses DIV2K and LSDIR with Real-ESRGAN’s higher-order degradation pipeline, while testing covers synthetic DIV2K/LSDIR patches and real-world RealSR and DRealSR.HQ and LQ resolutions are standardized to 512 × 512 and 128 × 128, respectively.
  • Experimental Setup: ASASR is evaluated with reference-based distortion and perceptual metrics alongside no-reference quality metrics, using a range of GAN-based and diffusion-based generative baselines.The reported metrics include PSNR, SSIM, LPIPS, DISTS, MANIQA, MUSIQ, and CLIPIQA.
  • Quantitative Comparisons: ASASR achieves superior perceptual scores on synthetic datasets, top SSIM ranks, dominant no-reference scores on real-world benchmarks, and leading full-reference performance.Figure 6 reports the lowest LSD of 27.35, with minimal residuals relative to the ground-truth spectral distribution.
  • Qualitative Comparisons: Qualitative comparisons show sharper geometries and facial details without baseline aliasing, authentic textures under unknown degradations, and effective artifact suppression.The spectral comparison further shows minimal residuals and the lowest Log-Spectral Distance.
  • User Study: In a study of 50 participants and 64 test images, ASASR achieves a Top-1 ratio of 91.1% when ranked for visual naturalness and fidelity.Participants ranked ASASR against baselines based on visual naturalness and fidelity.
  • Downstream Tasks Evaluation: ASASR achieves SOTA performance across downstream detection, instance segmentation, semantic segmentation, and OCR metrics, indicating preservation of high-level semantic structures.Evaluations use COCO, ADE20K, and ICDAR 2024 Occluded RoadText with Mask R-CNN, SegFormer-B5, and PaddleOCR v3.
  • Ablation Study: Ablations show that the full method outperforms variants across perceptual, quality, and semantic metrics, while Sobolev Guidance significantly surpasses Euclidean Guidance.The combined alignment strategy improves perceptual quality without sacrificing reconstruction fidelity and enhances downstream semantic integrity.
  • Sensitivity Analysis: Sensitivity analysis selects s = 1.5 as the equilibrium between fidelity and realism because s ≥2 improves reference-based metrics through smoothing but erases high-frequency textures.Stronger regularity degrades perceptual quality by removing high-frequency details.

6. Related Work

Image super-resolution has evolved from GAN-based training toward large-scale generative priors, encompassing diverse architectures and DPO-style adaptations. However, DP2O-SR’s heuristic IQA aggregation lacks a principled connection to the natural image manifold’s geometry.

  • Generative priors: Image super-resolution has shifted from GAN-based training toward large-scale generative priors for blind and real-world restoration.The cited generative priors are described as providing stronger natural image priors.
  • Generative priors: The modern paradigm includes Diffusion Transformers, autoregressive models, and DPO-style adaptations designed to better match human or perceptual preferences.The passage identifies these as diverse architectures or adaptation strategies within the broader generative-prior paradigm.
  • Preference optimization: DP2O-SR steers generation through aggregated IQA metrics, but its heuristic optimization lacks a principled theoretical foundation.The proxy objective is described as disconnected from the intrinsic geometric structure of the natural image manifold.

7. Conclusion

ASASR bridges isotropic generative priors and the natural image manifold through Sobolev Spectral Rectification. It uses Riemannian geometry and a Riesz-grounded adversary to promote perceptually plausible, structurally faithful restoration.

  • ASASR bridges the spectral gap between isotropic generative priors and the natural image manifold.
  • Sobolev Spectral Rectification constrains optimization within Riemannian geometry that respects natural images’ characteristic spectral decay.
  • A Riesz-grounded adversary uses worst-case Sobolev gradients to identify and rectify structural failures.

Impact Statement

The work presents a robust framework for high-fidelity image super-resolution with potential applications in consumer imaging and cloud-based restoration. It targets spectral artifacts, hardware-limited mobile imaging, and compressed user-generated content.

  • Consumer Imaging: The framework is designed for high-fidelity image super-resolution and has significant potential for consumer imaging deployment.The passage characterizes the method as robust and highlights consumer imaging as a deployment context.
  • Mobile Imaging: By mitigating spectral artifacts, the method helps mobile imaging systems address digital zoom and low-light noise limitations.The stated hardware limitations are digital zoom and low-light noise.
  • Cloud-Based Restoration: The approach is positioned as a cloud-based tool for revitalizing compressed uploads as user-generated content platforms expand.The passage gives photo-logs as an example of such platforms and links the use case to broader access to professional-grade imaging.

A. Derivation of the S-DPO Objective … B. Proof of Proposition in Adversarial Manifold Guidance

The section derives S-DPO from a continuous-time velocity-space policy, then replaces isotropic Euclidean geometry with Sobolev spectral geometry. This yields frequency-aware regularization that weights deviations according to spectral importance while retaining preference optimization.

  • A.1. Continuous-Time Velocity-Space Formulation: The derivation models Flow Matching as a probabilistic policy on the tangent bundle, avoiding vanishing SNR from discrete SDE approximations.The policy is defined over instantaneous velocities rather than noisy state transitions.
  • A.1. Continuous-Time Velocity-Space Formulation: Standard Flow Matching uses the ODE dxt = v(xt, t)dt and targets the conditional vector field ut(x|x1) = (x1 −xt)/(1 −t).The instantaneous velocity policy is centered at the model prediction vθ, with ut serving as the deterministic optimal action.
  • A.1. Continuous-Time Velocity-Space Formulation: Comparing optimal-velocity likelihoods under the policy and reference reduces the log-likelihood ratio to differences of squared Euclidean norms.Gaussian normalization constants cancel during expansion.
  • A. Derivation of the S-DPO Objective: Because the target velocity is deterministic, the residual-based objective remains numerically stable as ∆t →0 and recovers the ℓ2 preference objective.The residual is γψ := vψ(xt) −ut, with ut derived from boundary conditions.
  • A.2. Sobolev Spectral Rectification and S-DPO: Sobolev Spectral Rectification replaces isotropic variance η2I with structured spectral covariance Σs to mitigate the frequency-agnostic nature of the ℓ2 norm.This defines the frequency-aware extension of the S-DPO loss.
  • A.2. Sobolev Spectral Rectification and S-DPO: The policy and reference share support and curvature under Σs, so S-DPO penalizes deviations using spectrally weighted reference distances rather than Euclidean distance.The resulting likelihood ratio uses Mahalanobis distances and feeds the standard DPO logistic preference structure, with winners preferred over losers.

B.1. Proof of Proposition 1: Sobolev Geometry of Hard Negatives … C. More Experiments

The proofs derive Sobolev-constrained hard negatives as spectrally preconditioned descent directions and show that a sufficiently expressive parametric adversary recovers them. Capacity experiments further indicate gains from larger LoRA adapters before performance saturation.

  • B.1. Proof of Proposition 1: Sobolev Geometry of Hard Negatives: The adversary finds Sobolev-bounded perturbations that minimize Euclidean residual energy, targeting artifacts in the model’s null space while inducing structural deviation.The constraint is ∥δt∥Hs ≤ εt, and the objective is JL2.
  • B.1. Proof of Proposition 1: Sobolev Geometry of Hard Negatives: The constrained optimization reduces to minimizing the Euclidean gradient’s linear alignment under a Sobolev constraint, yielding a closed-form optimal perturbation through an active-boundary solution.The derivation uses the first-order Taylor expansion, Lagrangian optimization, and the active condition ∥δ∗t∥Hs = εt.
  • B.1. Proof of Proposition 1: Sobolev Geometry of Hard Negatives: Σs acts as a Sobolev natural-gradient preconditioner, filtering incoherent high-frequency components into structurally consistent degradations rather than imperceptible noise.This supports a parametric adversary that approximates recurring artifacts and texture distortions to penalize hallucinations.
  • B.2. Proof of Proposition 2: Theoretical Consistency of the Parametric Adversary: The parametric adversary translates the state-space Sobolev direction into a velocity-space correction using Euler integration and a bounded variational optimization.The formulation assumes sufficient representational capacity and absorbs Δt into the effective step size.
  • B.2. Proof of Proposition 2: Theoretical Consistency of the Parametric Adversary: The velocity-based solution is identical to Proposition 1’s state perturbation, confirming that bounded velocity-network optimization implicitly learns the optimal Sobolev-preconditioned descent direction.This establishes theoretical consistency between the functional optimum and its parametric realization.
  • B.3. Empirical Examination of the Capacity Assumption: Increasing adversary capacity through larger LoRA adapters produces clear performance gains from very small sizes, after which performance largely saturates.The experiment varies LoRA size to examine the practical reasonableness of the sufficient-capacity assumption.

C.1. More Implementation Details

The implementation uses memory-efficient training and a 28-step inference schedule, with DCT-based spectral operations and Dual-LoRA reference-policy management. On an NVIDIA A800, it achieves high hardware utilization and practical inference efficiency.

  • Implementation: BF16 mixed-precision training with gradient checkpointing is enabled on the FLUX.1-dev backbone, and inference uses 28 sampling steps to balance quality and latency.These choices target training throughput, memory efficiency, and inference latency.
  • Implementation: Adversary performance improves with capacity in the small-adapter regime and largely saturates beyond moderate sizes.The comparison is reported under different LoRA sizes.
  • Implementation: A Dual-LoRA strategy keeps the SFT adapter MSFT frozen for the reference distribution while updating a separate zero-initialized adapter MDPO.This avoids duplicating FLUX.1-dev’s massive parameters for the DPO reference policy.
  • Implementation: The spectral operator uses DCT-II with symmetric reflection, implicitly enforcing Neumann boundary conditions and avoiding boundary discontinuities associated with the periodic DFT assumption.The design preserves boundary continuity and regularity consistent with the Sobolev norm ∥· ∥Hs.
  • Efficiency: 46.6% MFU, 33.02 GB peak VRAM usage, 96.2% memory efficiency, and 19.60 seconds average latency per image are reported on an NVIDIA A800.Metrics are averaged over 100 inference runs excluding warm-up.

C.2. More Quantitative Experiments · C.3. User Study · C.4. More Super-Resolution Experimental Results

Across additional backbones and challenging real-world datasets, ASASR maintains competitive or superior restoration quality beyond its training distribution. A 50-participant user study and extensive visual task evaluations further show strong perceptual preference, structural fidelity, and semantic preservation.

  • C.2. More Quantitative Experiments: ASASR consistently improves over corresponding SFT baselines with both SD1.5 and SDXL, indicating effectiveness beyond the FLUX backbone.The experiments evaluate generality across different generative backbones.
  • C.2. More Quantitative Experiments: On RealLQ250 and Bringing Old Films Back to Life, ASASR achieves competitive or superior performance across multiple non-reference image quality metrics.These datasets represent challenging real-world degradations outside the training distribution.
  • C.3. User Study: The user study assessed perceptual realism and semantic fidelity using 64 low-quality images, ASASR, six baselines, and 50 participants.Compared methods were DiT4SR, DP2OSR, SUPIR, SeeSR, StableSR, and RealESRGAN.
  • C.3. User Study: Vote Percentage measures first-place frequency, while Top-K Ratio measures how often method i appears within the top-k rankings.The Top-K computation uses evaluation groups, method rankings, and an indicator function.
  • C.3. User Study: 91.1% of first-place votes went to ASASR, which outperformed all baselines by a substantial margin in the Top-1 Ratio analysis.ASASR also remained consistently among the top choices across all k levels.
  • C.4. More Super-Resolution Experimental Results: Visual comparisons on synthesis datasets show superior structural fidelity for architectural geometries, legible text, and natural textures, while avoiding distortions and hallucinations.The comparisons include state-of-the-art GAN-based and diffusion-based methods.
  • C.4. More Super-Resolution Experimental Results: On real-world scenes, ASASR restores intricate high-frequency details while balancing noise suppression and perceptual photorealism.Examples include feather textures and distant architectural features, contrasting with over-smoothed or hallucinated baseline outputs.
  • C.4. More Super-Resolution Experimental Results: Downstream evaluations show clearer OCR text, more precise COCO detections and segmentation masks, and cleaner ADE20K semantic maps after ASASR restoration.These results reflect preserved text characters, object boundaries, and semantic regions with fewer artifacts.
Loading 2605.23264v2…