PaperlayerView exact revision on arXiv

Source-linked AI summary

Super-Resolution in The Right Latent Space: A Frozen Vision-Foundation Substrate

Wanzhou Lei, Cuifeng Shen, Yanjin He, Maohua Li, Hua Yuan, Per-Olof Persson, Tao Lan, Kan Liu, Hanlin Tang

arXiv:2610.04781v2cs.CV

TL;DR

Real-world super-resolution must map degraded images to faithful points on the clean-image manifold, but existing pixel-space and VAE-latent approaches lack the desired proximity and information hierarchy. The paper uses fused layers of a frozen DINOv3-L encoder as the substrate for one-pass RAESR, which attains the best reported fidelity–perception and efficiency–perception trade-offs among the evaluated methods across four benchmarks.

  • Problem

    Real-world super-resolution needs to map degraded embeddings to faithful points on the clean-image manifold, while existing pixel-space and VAE-latent methods use reconstruction-oriented spaces that lack proximity to the manifold and semantic guidance.

    Source: §1 · pp. 1–2

  • Method

    The paper uses fused layers of a frozen DINOv3-L encoder as the substrate for a one-pass 415M-decoder restorer, RAESR.

    Source: §1 · p. 2

  • Results

    Across four benchmarks, RAESR ranks first among twelve methods on NIQE, MANIQA, and DISTS and second on MUSIQ, LPIPS, and TOPIQ, with the best fidelity–perception and efficiency–perception trade-offs at 37 ms per image on one H20.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    The paper argues that choosing a suitable latent substrate should be central to real-world super-resolution research.

    Source: §1 · p. 2, §6 · p. 10

    RAESR has no mechanism to spend more steps on hard images, and the substrate-suitability claim is characterized on DINOv3-L with only one alternative substrate tested.

    Source: §6 · p. 10

Abstract

from arXiv · show

In an image latent space, the embeddings of high-resolution, natural, and sharp images form a manifold. Degradation of high-resolution images pushes their embeddings off this manifold. Real-world super-resolution (SR) then becomes the task of mapping the degraded embedding back onto this manifold --- not anywhere on the manifold, but to the point that preserves what the input still carries, both its semantics and pixel details. Every published method implements this mapping in a reconstruction-oriented latent space or pixel space. We claim these spaces are the wrong substrates for SR. Low-resolution and degraded images are embedded far from the manifold, making the mapping difficult and expensive. The lack of semantic information in these substrates also makes it difficult to navigate to the faithful point on the manifold, causing severe hallucination when degradation is heavy. Thus, restoring in a suitable latent space is crucial to the SR task. We show that the latent space of 23 fused layers of a frozen DINOv3-L is one such space that makes the SR task easier. Degraded images are embedded near the manifold. Moreover, this substrate contains a hierarchy of information, from pixel record to degradation robust semantics, guiding the model to find the faithful point on the manifold. On this substrate, a 415M decoder is trained under reconstruction and adversarial objectives to map the degraded embeddings back and decode to pixel space in one pass. The resulting model, RAESR, attains the best fidelity--perception trade-off among state-of-the-art adversarial and diffusion-based restorers on RealSR, DRealSR, LSDIR and DIV2K-Val, at 37 ms per 512 by 512 image on a single H20 GPU. Swapping the substrate for a VAE latent under an identical recipe loses on every metric.

1 Introduction

Real-world super-resolution maps degraded-image embeddings back to the clean-image manifold, but reconstruction-oriented spaces make that mapping difficult and weakly guided. The paper proposes fused DINOv3-L layers as a more suitable substrate and builds RAESR, which achieves strong benchmark trade-offs in one decoder pass.

1 Introduction

  • Real-world super-resolution maps degraded embeddings back to the clean-image manifold, aiming for the point faithful to the input’s semantics and surviving detail.A suitable substrate should place degraded images near the manifold and provide information from pixels through semantics to guide the mapping.Source: §1 · p. 1
  • Pixel-space and VAE-latent methods preserve degradation and lack semantic guidance, making the mapping difficult and potentially leading to hallucinations.Lightweight GAN restorers can miss the manifold, while methods using visual generative priors incur large models and slow inference.Source: §1 · pp. 1–2
  • The fused DINOv3-L latent places degraded images near the manifold and preserves a hierarchy from shallow-layer pixel records to deep, degradation-robust semantics.The paper argues this makes a one-step map easier to learn and guides it toward the faithful point.Source: §1 · p. 2
  • The paper builds RAESR as a one-pass restorer and argues that future super-resolution research should choose substrates suited to the task.Its stated substrate criteria are proximity of degraded embeddings to the manifold and information at all levels to guide reconstruction.Source: §1 · pp. 2–3
  • Across four benchmarks, RAESR ranks first among twelve methods on NIQE, MANIQA, and DISTS and second on MUSIQ, LPIPS, and TOPIQ, with the best fidelity–perception and efficiency–perception trade-offs at 37 ms per image on one H20.The comparison averages results across the four benchmarks.Source: §1 · p. 2, Figure 2 · p. 3

2 Related Work

Prior real-world super-resolution methods generally retain reconstruction-oriented pixel or VAE spaces, while workarounds address their defects without changing the substrate. Representation autoencoders offer a frozen-encoder alternative developed for generation and clean reconstruction, which this paper carries over to restoration.

2 Related Work

  • One-pass adversarial restorers in reconstruction-oriented spaces are small and fast but produce blurry outputs, while diffusion methods use large models and tens of steps.Other approaches distil diffusion schedules or train short chains, but the mapping remains long and weakly guided in the same substrate.Source: §2 · p. 3
  • Prior work mitigates distance or missing guidance with encoder fine-tuning, pixel-space restoration, semantic conditioning, or DINOv2 features, but retains the reconstruction-oriented substrate.This paper takes the contrasting route of changing the substrate.Source: §2 · p. 3
  • Representation autoencoders use a frozen self-supervised encoder with a trained decoder; RAEv2’s multi-layer fusion is adopted here for restoration rather than generation and clean reconstruction.Source: §2 · p. 3

3 Method

The method fuses DINOv3-L layers into a frozen substrate and trains a decoder with reconstruction and adversarial objectives, supported by a layer-wise critic.

3.1 Fused Layers of DINOv3-L as the Latent

  • 40.95 dB to 28.09 dB reconstruction PSNR and 6.5% to 83.6% linear-probe accuracy mark a hierarchy from pixel detail to semantics across DINOv3-L layers.The fused latent averages the last 23 layers plus the final layer’s spatial average, leaving out the first layer.Source: §3.1 · p. 4, Figure 3 · p. 4
  • Most layers remain linearly recoverable from fused z, with R2 around 0.8 or higher and shallow layers near 0.9 on clean and training-degraded inputs.This linear recovery is a lower bound on what the nonlinear decoder can recover.Source: §3.1 · p. 4

3.2 One decoder pass

  • A trainable decoder carries the restoration burden over the frozen substrate, using 28 transformer layers to predict pixels from its token grid.The decoder has 415M parameters and predicts 162×3 pixels per token before unpatchifying.Source: §3.2 · p. 4

3.3 Training objectives

  • L1 and LPIPS supervise faithfulness, while the adversarial term supplies realistic texture instead of regressing toward blurred averages.The loss uses λp = 1.0 for VGG16 LPIPS and λadv = 0.5 wherever the adversarial term is active.Source: §3.3 · p. 5
  • Training moves from bicubic-downsampled LSDIR images, to strong Real-ESRGAN degradation with the critic, then to mixed-severity degradations.The three-stage schedule finishes by aligning training with deployment.Source: §3.3 · p. 5

3.4 Adversarial training

  • A frozen DINOv3-B critic uses 12 trainable layer heads to score each token, comparing restoration and ground-truth logits at matching positions.Judgments are dense rather than pooled, and the reference score sets the realism target for each location.Source: §3.4 · p. 5, Figure 4 · p. 6

4 Experiments

The experiments compare twelve public methods on four real-world ×4 SR benchmarks and report RAESR’s benchmark performance, efficiency, qualitative restorations, and human preferences. Across the reported metrics, RAESR achieves leading fidelity–perception trade-offs and observers prefer it to four competitive baselines.

4.1 Protocol

  • Evaluation covers four unseen benchmarks and twelve public methods, using official checkpoints and defaults; latency is measured at batch 1 on one H20 GPU.The benchmarks are RealSR, DRealSR, LSDIR and DIV2K-Val; nine metrics are averaged over them.Source: §4.1 · p. 6

4.2 Main results

  • Across four benchmarks, RAESR ranks first on NIQE, DISTS and MANIQA and second on LPIPS, MUSIQ and TOPIQ; it has the best fidelity–perception trade-off.Table 1 reports the ×4 real-world SR results macro-averaged over the four benchmarks.Source: §4.2 · p. 6, Table 1 · p. 7
  • RAESR is 1.6× faster than AdcSR and 59–514× faster than 20–50-step methods; TSD-SR is 3.1× larger and 2.8× slower.Source: §4.2 · p. 8
  • In qualitative comparisons, RAESR alone preserves the boat canopy, the background building, and the fox-nose fur direction.Figure 5 shows full frames and boxed regions enlarged at 4× nearest-neighbour magnification, with RAESR in red.Source: Figure 5 · p. 7, §4.2 · p. 8
  • RAESR-Distilled largely preserves teacher performance while reducing inference parameters to 502M and latency to 30.0 ms per image.The full RAESR uses 719M inference parameters and runs at 37.2 ms per image.Source: §4.2 · p. 8
  • Human observers prefer RAESR to each of four baselines, including a weak win over TSD-SR.The study used 100 images, sampled uniformly from the four benchmarks, with anonymous restorations and randomized left/right assignment.Source: Human preference study · p. 8, Table 2 · p. 8

5 Analysis

The analysis tests whether RAESR’s behavior comes from the DINOv3-L substrate: degraded images lie nearer the clean-image manifold, and substrate information at multiple levels guides reconstruction. A controlled substrate swap and layer-band interventions probe those claims.

5 Analysis

  • The analysis tests the hypothesis that RAESR’s performance comes from a substrate that places degraded images near the manifold and guides mapping at every level, rather than from the decoder.Source: §5 · p. 8

5.1 Degraded images are embedded near the manifold

  • Across four metrics and 2,000 LSDIR images, decoded points along the fused-latent line recover clean-image statistics faster than points along the SD3 VAE line.The endpoint-matched curves separate most in the middle, indicating degraded embeddings are relatively closer to both the manifold and the faithful high-resolution embedding in the fused latent.Source: §5.1 · p. 8, Figure 6 · p. 9

5.2 The decoder relies on every level of the substrate

  • Removing the shallow eight-layer band harms fidelity more, while removing the deep eight-layer band harms perceptual metrics more; removing the broadcast term widens both effects.The sample interventions show lost fidelity to input pixels without the shallow band and generic texture synthesis without the deep band.Source: §5.2 · p. 9, Figure 7 · p. 9

5.3 The semantics come from the substrate, not the decoder

  • With the decoder and recipe held largely fixed, replacing DINOv3-L with an SD3 VAE loses on all nine metrics; heavy degradation yields sharp wrong content or images that are not clean and natural.Figure 8 presents the substrate comparison.Source: §5.3 · p. 9, Figure 8 · p. 10

6 Conclusion and Limitations

The paper argues that latent-substrate choice matters for real-world SR and reports RAESR’s trade-offs and human preference results. Its stated limitations are the one-pass design and evidence centered on one encoder family and one alternative substrate.

6 Conclusion and Limitations

  • The authors argue that fused layers of frozen DINOv3-L provide a suitable substrate, and RAESR achieves the best fidelity–perception and efficiency–perception trade-offs among twelve evaluated methods.The conclusion also reports that human observers prefer RAESR to four evaluated models.Source: §6 · p. 10
  • RAESR has no mechanism to trade additional compute for quality on hard images, and substrate claims cover fused DINOv3-L plus one alternative, leaving other encoders open.Source: §6 · p. 10

A Human preference study

  • The human preference study uses 100 randomly sampled images, anonymous restorations, randomized left/right assignment, and asks observers to judge faithfulness and image quality together.Participants judge 25 distinct images each; the ground truth is shown deliberately.Source: A Human preference study · p. 14

B Evaluation details and per-benchmark results

B.1 Evaluation protocol

  • PSNR and SSIM are computed on the Y channel, and deterministic metric computation means scored checkpoints reproduce their values to machine precision.Single entries carry no evaluation error bar.Source: §B.1 · p. 14

B.2 Baseline configurations and training signals

  • Baseline rows use official checkpoints and defaults, and no method—including RAESR—trains against the reported no-reference IQA metrics.The appendix compiles baseline training-loss signals from official papers and, where available, official code.Source: §B.2 · p. 14

B.3 Per-benchmark results

RAESR retains strong per-benchmark perceptual rankings, and the appendix examines its fused score, channel-mean alignment, and the information carried by the frozen encoder’s layers. The layer analysis finds both redundancy between neighboring layers and substantial recoverability from the fused latent.

B.3 Per-benchmark results

  • Excluding LSDIR, RAESR leads on NIQE, DISTS and MANIQA and ranks second on MUSIQ and TOPIQ across the remaining benchmarks.The per-benchmark results cover LSDIR, RealSR, DRealSR and DIV2K-Val.Source: §B.3 · p. 14, Table 4 · p. 15, Table 5 · p. 16

B.4 The fused perceptual score of Figure 2

  • The Figure 2 perceptual score averages six min–max-normalized metrics, with DISTS and NIQE negated so higher scores are better.Each metric contributes equally to the bounded [0, 1] score.Source: §B.4 · p. 15
  • Replacing min–max normalization with z-scores preserves the ranking of all 14 methods; mean metric ranks retain the top position and change only adjacent mid-field pairs.Across the six metrics, their spread is 23–58% of the metric mean, so the passage reports no near-tie normalization failure mode.Source: §B.4 · p. 15

B.5 Channel-mean alignment

  • Channel-mean alignment adds 0.36 dB PSNR to the teacher and 0.99 dB to the student, with no measurable inference cost.Teacher MUSIQ rises 0.06, while no other metric changes by as much as 0.001 in either direction.Source: §B.5 · p. 16

C The frozen encoder: full geometry

  • The frozen-encoder analysis measures normalized patch-token outputs from DINOv3-L’s 24 blocks, excluding CLS and register tokens; the fused latent averages layers 2–24 and adds layer 24’s spatial mean.Layer 1 is not included in the latent.Source: C The frozen encoder: full geometry · p. 16

C.1 Layer disentanglement and redundancy

  • Layer overlap falls below chance at an 11-layer separation and reaches −0.60 between the first and last layers, while adjacent layers have overlap 0.64 and CKA 0.95.Beyond 15 layers apart, CKA flattens at 0.14–0.16; the analysis characterizes distant layers as disentangled and neighboring layers as redundant.Source: §C.1 · pp. 16–17, Figure 9 · p. 17

C.2 Recovery of each layer from the fused latent

  • A linear map from the fused latent recovers layer tokens with R2 of 0.695–0.866 on clean inputs and 0.646–0.893 on degraded inputs.On degraded inputs, layer 1 recovery rises from 0.695 to 0.893 while layer 24 falls from 0.839 to 0.646; the linear scores are lower bounds on what the decoder can extract.Source: §C.2 · p. 18
  • The best single-layer predictor is always adjacent, with R2 of 0.865–0.988 on clean inputs and 0.783–0.998 on degraded inputs, exceeding fused-latent recovery at every layer.The passage attributes the small loss from averaging to neighboring-layer redundancy; recovery also tracks layer dimensionality, which rises from 9 principal components at layer 1 to 404 at layer 24.Source: §C.1 · p. 17, §C.2 · p. 18

D Latent-line geometry: protocol, thresholds and replications

The latent-line experiments compare decoded interpolations between degraded and clean embeddings under matched protocols and endpoints. Replications and controls distinguish progress along the clean-image direction from arbitrary latent movement, decoder robustness, and reaching a natural but incorrect target.

D Latent-line geometry: protocol, thresholds and replications

Endpoint matching and the SD2.1 replication

Random-direction control

  • The random-direction control restores nothing in the fused latent: NIQE shifts from 9.93 to 9.95 and LPIPS from 0.823 to 0.801 over the walk.The control trajectory is worse than the true path on LPIPS, DISTS, PSNR and SSIM, with paired probability 1.00.Source: D Latent-line geometry: protocol,… · p. 19

Noise augmentation is not the mechanism

  • Noise-smoothed-latent robustness is direction-blind and cannot explain why movement toward the clean embedding improves every metric family.The passage contrasts that invariance with the changing decode along the true line.Source: D Latent-line geometry: protocol,… · pp. 19–20

Wrong-target control: membership is not fidelity

  • Both decoders make wrong-target interpolations natural at α=1, but LPIPS to the true image never falls below 0.74.The passage distinguishes reaching the manifold from reaching the correct point on it.Source: D Latent-line geometry: protocol,… · p. 20

E Decoder channel ablation

Inference-time channel ablations test which averaged DINOv3 layers and broadcast information matter to fidelity and perception. The appendix also describes the adversarial critic schedule and a distilled encoder that reuses the decoder unchanged.

E Decoder channel ablation

  • Removing the shallow band yields F = +40.2% against P = +8.2%, while removing the deep band yields F = −2.1% against P = +7.2%.These separations lie outside the random-band range and the crossing holds on each of the four benchmarks; Table 9 summarizes the ablation.Source: E Decoder channel ablation · p. 20
  • Removing the deep band and broadcast term gives F = −1.3% against P = +12.1%, a 13.4-point separation that exceeds the deep-band-only separation.PSNR improves by 0.76–1.79 dB on every benchmark, and SSIM improves by 0.038.Source: E Decoder channel ablation · pp. 20–21

F The adversarial term: schedule

  • The final-stage critic sees a 496×496 center crop offset from the decoder’s 16px token grid, while other losses use the full 512px image.The schedule also uses two critic updates per generator step at a 6× learning rate, then ramps in the adversarial weight after warming the critic at zero.Source: F The adversarial term: schedule · p. 21

G RAESR-Distilled

  • RAESR-Distilled replaces the frozen 303M encoder with frozen DINOv3-S/16+ and a learned projector, reusing the released decoder without retraining it.The replacement encoder is 28.7M parameters and the projector is 57.8M.Source: G RAESR-Distilled · p. 21

H The substrate ablation of §5.3

The substrate ablation replaces frozen DINOv3-L with a frozen SD3 VAE while keeping the decoder recipe largely matched. The reported reference-perceptual gap is largest on LSDIR, the most heavily degraded set.

H The substrate ablation of §5.3

  • The ablation freezes the SD3 VAE and changes only the decoder input projection’s shape and initialization, warm-starting every other weight from the released decoder.Both arms use the final 600-step EMA and are scored before channel-mean alignment.Source: H The substrate ablation of §5.3 · p. 21
  • The SD3 VAE substrate’s reference-perceptual gap is largest on LSDIR, with LPIPS +0.081 and DISTS +0.056.LSDIR is identified as the most heavily degraded set; the passage describes the sweep as total.Source: H The substrate ablation of §5.3 · p. 21
  • The reported recipe deviations are that stage 1 used the full 166,994-image training pool at multiple resolutions and stage 2 used global batch 15 rather than 16.Source: H The substrate ablation of §5.3 · p. 21

I Training details and protocols

The training protocol freezes the DINOv3-L encoder while fitting the decoder through staged reconstruction and adversarial training; the sample gallery presents outputs across four benchmarks.

I Training details and protocols

  • The DINOv3 substrate leads on all nine metrics in the encoder-substrate ablation under the released recipe.The table caption identifies the comparison but does not provide metric values.Source: Table 10 · p. 22
  • Training uses LSDIR for stages 1–2 and a row-weighted LSDIR, FFHQ-1024, and Aesthetic-4K manifest for stage 3, with benchmarks held out from training.Stage 3's manifest contains 1,421,913 rows over 166,994 unique images; RealSR, DRealSR, and DIV2K contribute no training images, and LSDIR evaluation uses its official 250-image validation split.Source: I Training details and protocols · pp. 21–22
  • Three stages progress from bicubic reconstruction initialization to strong-degradation training and a released mixed-severity finish.Stage 2 switches on the adversarial term after 2,000 reconstruction-only steps; stage 3 uses strong, medium, and mild degradation at probabilities 0.30, 0.50, and 0.20, respectively, and releases the EMA at step 600.Source: I Training details and protocols · pp. 22–23
  • Distillation freezes the encoder and decoder while training only a 57.8M-parameter projector in two stages.The second stage combines latent MSE with image-output L1 and LPIPS losses, and the released student is the final step-20,000 checkpoint, also best on the held-out validation objective.Source: I Training details and protocols · p. 23

J Sample gallery

Related papers

Loading 2610.04781v2…