PaperlayerView exact revision on arXiv

Source-linked AI summary

SPARK: Input-Conditioned Sparse Activation Modulation for Frozen DiT-based Super-Resolution

Federico Putamorsi, Leonardo Zini, Marcella Cornia, Lorenzo Baraldi

arXiv:2609.03813v1cs.CV

TL;DR

Real-world DiT-based super-resolution leaves open whether perceptual quality requires modifying the full network or can be controlled through a small structured activation subset. SPARK selects dominant channels online and uses an input-conditioned predictor to apply bounded affine modulation while freezing the backbone and VAE. Across three backbones and multiple benchmarks, it consistently improves fidelity and perceptual quality, while its scope is limited to settings with sufficiently sparse dominant activations.

  • Problem

    It is unclear whether perceptual quality in DiT-based SR requires modifying the full network or can be controlled through a small structured subset of representations.

  • Method

    SPARK uses online dominant-channel selection and an input-conditioned predictor of bounded affine parameters, optimizing only the predictor while freezing the SR backbone and VAE.

  • Results

    Across three DiT-based SR backbones and multiple benchmarks, SPARK consistently improves fidelity and perceptual quality while modulating only eight channels per stream and block.

  • Takeaways & Limitations

    Sparse intervention in dominant activation channels is an effective alternative to weight-space adaptation for the evaluated DiT-based SR models.

  • Takeaways & Limitations

    SPARK may be less effective when activation magnitudes are distributed uniformly, and its backbone-specific selection must be repeated for new architectures or checkpoints.

Abstract

from arXiv · show

Real-world image super-resolution (SR) increasingly relies on Diffusion Transformer (DiT) backbones, whose internal activations can be dominated by a small number of massive channels. Yet improving perceptual quality in these models still typically requires fine-tuning the network or attaching additional adapters, leaving this structured activation space largely unexplored for adaptation. We investigate whether dominant channels can instead serve as a compact adaptation interface for frozen DiT-based SR models. We first characterize their behavior in pretrained SR backbones and show through controlled interventions that they strongly affect reconstruction quality. Building on this observation, we introduce SPARK, a lightweight input-conditioned controller that predicts bounded per-channel affine transformations for only the selected channels, while keeping the SR backbone and VAE frozen. Dominant channels are identified through an online activation-ranking procedure, and only a small predictor conditioned on the low-resolution VAE latent is optimized. Experiments on three DiT-based SR backbones across DIV2K, RealSR, and DRealSR show consistent gains in both fidelity and perceptual quality while modulating only eight channels per stream and block. Controlled comparisons further show that these gains cannot be explained by parameter budget or access to the selected channels alone.

1. Introduction

SPARK asks whether perceptual quality in DiT-based super-resolution can be controlled through a small, structured subset of activations rather than full-network modification. It identifies dominant channels as functionally important and modulates them with a lightweight predictor while freezing the SR backbone and VAE.

  • Dominant channels are highly activated and controlled ablations show they strongly affect reconstruction quality.The analysis distinguishes statistical prominence from functional importance.
  • SPARK modulates selected dominant channels with an image-conditioned predictor while keeping the DiT-based SR backbone and VAE frozen.The predictor maps low-resolution VAE features to bounded affine parameters for selected channels.
  • The method combines online, stability-terminated channel selection with bounded affine modulation of the selected channels.This design targets a compact activation-space interface rather than weight-space adaptation.
  • Experiments cover three DiT-based SR backbones and multiple benchmarks, evaluating both fidelity and perceptual-quality metrics.Controlled comparisons isolate channel selection, parameter budget, and modulation mechanism.

2. Related Work

Prior super-resolution work moved from distortion-focused models toward generative diffusion, Transformer, and flow-matching backbones to improve perceptual realism and efficiency. Massive activations provide related context for examining whether a sparse activation subset can serve as an adaptation interface.

  • Early SR methods targeted synthetic degradations with pixel-wise losses, but often generalized poorly to real-world noise, blur, and compression.
  • Diffusion-based SR improves perceptual quality and texture realism, while one- and few-step methods address the inference cost of multi-step denoising.
  • Recent SR systems use Transformer-based generative priors, including DiT, for global modeling, scalability, conditioning, and efficient restoration.
  • Massive activations are unusually large responses concentrated in a small number of channels and can dominate feature propagation and attention.Related studies associate them with texture synthesis, spatially coherent information, and trade-offs involving semantic consistency.
  • Figure 2 frames dominant-channel analysis through activation concentration, ablation sensitivity, metric degradation, and qualitative restoration effects.

3. Dominant Channels as an Adaptation Subspace

The analysis tests whether dominant channels form a compact and intervention-sensitive adaptation subspace in pretrained DiT-based SR models. Their activation energy is highly concentrated, and removing them harms reconstruction more than removing random or low-ranked channels.

  • Activation magnitudes are highly uneven, with a small subset much larger than the majority across DiT blocks and streams.Figure 2(a) visualizes this skewed activation landscape.
  • The top 32 channels, about 2% of the channel dimension, have a median Gini coefficient of 0.62 across blocks.
  • For K = 8, dominant channels contain 85.6% of encoder-stream energy and 68.8% of hidden-stream energy while occupying 0.5% of channels.
  • Removing top-ranked channels sharply degrades reconstruction quality, whereas random removal has moderate effects and bottom-ranked removal has minor effects.The comparison averages SSIM, LPIPS, LIQE, MANIQA, MUSIQ, and CLIP-IQA changes across K.
  • Top-K ablation causes sharp, nearly monotonic degradation across perceptual and distortion-based metrics, with visible artifact growth as K increases.Even small perturbations reduce perceptual quality, while larger perturbations damage fine details and structure.
  • Dominant channels provide substantially higher leverage over reconstructed output than equally sized lower-ranked subsets.This asymmetry motivates using them as a compact adaptation interface.

4. Proposed Method

SPARK adapts frozen DiT-based SR models by selecting dominant channels and learning a lightweight predictor that applies bounded, input-conditioned affine modulation only to those channels. The method uses online channel ranking, low-resolution VAE-latent conditioning, and a frozen backbone to improve perceptual quality efficiently.

  • Phase 2: Image-Conditioned Predictor: The two-phase framework combines channel selection, input-conditioned modulation, and a training objective balancing LPIPS fidelity, LIQE perceptual quality, and total variation regularization.Only the predictor parameters are optimized during training.
  • Phase 1: Channel Selection via Activation-based Importance: SPARK selects dominant channels online using activation-based importance and applies modulation only within this sparse subspace.The ranking uses sequential mini-batch statistics and stops after the selected subset stabilizes, retaining K = 8 channels per stream and block.
  • Phase 2: Image-Conditioned Predictor: The predictor applies feature-wise affine transformations to selected hidden and encoder activations while the SR backbone and VAE remain frozen.The predicted parameters are input-dependent and constrainable to fixed intervals for stable modulation.
  • Phase 2: Image-Conditioned Predictor: A lightweight predictor maps pooled features from the low-resolution VAE latent to independent scale–shift parameters for every selected channel.The output dimensionality reflects one scale–shift pair for each selected channel across all blocks and streams.
  • Phase 2: Image-Conditioned Predictor: Despite modulating only K channels per stream and block, SPARK consistently improves perceptual quality over the frozen baseline.The result supports sparse activation control as an efficient adaptation mechanism that leaves the pretrained SR backbone unchanged.

5. Experiments

Experiments evaluate SPARK across three DiT-based SR backbones and synthetic and real-world benchmarks, showing broad gains from sparse activation modulation. Controlled studies isolate the effects of adaptation mechanism, channel choice, selected-channel count, perceptual objective, and computational cost.

  • Main results: SPARK improves 61 of 63 metric-dataset-backbone combinations, including all nine SSIM settings and seven LPIPS settings, across DRealSR, RealSR, and DIV2K-Val.The gains transfer from synthetic training degradations to both real-world benchmarks; TEASR improves 20 of 21 scores.
  • Alternative adaptation strategies: SPARK achieves the strongest MUSIQ, CLIP-IQA, TOPIQ, and LIQE scores among shared top-8-channel mechanisms, while LoReFT leads MANIQA and LoRA leads SSIM and LPIPS.The comparison uses the same selected channels for most alternatives and a comparable parameter budget for parameter-matched LoRA.
  • Channel selection: Top-activation channel selection at K = 8 yields the best MANIQA, MUSIQ, CLIP-IQA, TOPIQ, LIQE, and SSIM scores, while random selection gives the best LPIPS.The result identifies dominant channels as the most effective subspace for the stated perceptual SR objective.
  • Number of selected channels: Most performance gains are recovered by K = 8, after which increasing the selected-channel count produces smaller and metric-dependent improvements.MANIQA and MUSIQ continue improving gradually, CLIP-IQA benefits up to intermediate K values, and SSIM remains consistently high.
  • Perceptual objective: Replacing LIQE with MANIQA, MUSIQ, or TOPIQ during training retains strong cross-metric performance, including on perceptual metrics not directly optimized.The supplied results support robustness to the choice of perceptual training objective.
  • Efficiency and adaptation cost: SPARK keeps the DiT and VAE frozen and uses a 274K-parameter predictor, below the 361K parameter-matched LoRA baseline.Channel selection uses gradient-free forward passes and typically terminates after roughly 400 TSD-SR images, while Phase 2 takes about 2.5 hours on one NVIDIA L40S GPU.

6. Conclusion

The paper presents dominant activation channels as a sparse, intervention-sensitive interface for adapting frozen DiT-based SR models. SPARK uses this interface to improve fidelity and perceptual quality across multiple backbones and benchmarks without weight-space adaptation.

  • 6. Conclusion: SPARK improves fidelity and perceptual quality across three SR backbones and multiple benchmarks while keeping the backbone frozen.The conclusion frames sparse representation-space intervention as an alternative to weight-space adaptation.

Supplementary Material

Supplementary experiments describe the computational requirements of SPARK's two adaptation phases and inference across several backbones. The reported costs vary substantially with backbone and formulation.

  • Computational resources: Phase 1 uses a single partial forward pass without gradients, typically processing about 400 TSD-SR images in 8 minutes or 300 TEASR images in 5 minutes.Only per-channel mean activations are stored, producing negligible memory overhead.
  • Computational resources: Phase 2 runs on one 48 GB NVIDIA L40S GPU, with TEASR stored in FP8 and using about 26 GB peak memory.The same single-GPU setup is used for all baselines, including LoRA and other parameter-efficient variants.
  • Computational resources: Training and inference times vary by backbone: TSD-SR requires about 2.5 hours for training, TEASR 15 hours, and DiT4SR up to 36 hours.DIV2K inference takes approximately 30 minutes for TSD-SR, 1 hour for TEASR, and 7 hours for DiT4SR.

A.2. Multi-Seed Stability Analysis

Repeated TSD-SR experiments on DRealSR show low cross-seed variability, with performance trends and configuration rankings preserved across seeds.

  • The stability analysis repeats all TSD-SR experiments on DRealSR with three random seeds and reports averages across runs.
  • 0.0021 normalized mean standard deviation indicates low cross-seed variability across metrics and configurations.Variations remain small relative to differences between configurations and do not change their overall ranking.
  • Relative variability remains small for bottom-K, top-K, and random selection, despite greater stochasticity for random channel selection.The reported standard deviations are 0.0014 for bottom-K, 0.0022 for top-K, and 0.0027 for random selection.

B. Extended Activation Analysis

Extended analyses confirm that dominant channels are functionally important and that top-channel selection generally provides the strongest perceptual control, while bounded scale-and-shift modulation offers the best balance.

  • Zero Ablation: Zeroing top-ranked channels degrades every reported metric, progressively destroying fine detail and structure, unlike random or bottom-ranked ablations.
  • Channel Selection: Top-activation channels provide the best no-reference perceptual quality across three backbones, except for TOPIQ on DiT4SR, where bottom selection is marginally higher.All variants modulate eight channels per stream and block.
  • Channel Selection: Top-channel modulation favors perceptual quality, while less-active-channel intervention can favor fidelity metrics depending on the backbone.On TSD-SR and TEASR, less-active channels yield the best LPIPS; on DiT4SR, top selection is strongest on both fidelity metrics.
  • Selection Criterion: Mean absolute activation slightly outperforms standard deviation as the channel-selection criterion, improving six of seven metrics.It trails standard deviation only on MANIQA.
  • Modulation Parameterization: Combining scale and shift gives the most consistent modulation performance, while unconstrained outputs substantially degrade SSIM and produce woven high-frequency hallucinations.The bounded affine transformation is best or second best on every metric.

C.4. How Far Does the Effect Extend? A Sliding Window over Channel Ranks

A sliding-window study shows that channel importance is graded rather than strictly binary: top-ranked windows give the strongest aggregate perceptual results, while lower-ranked windows retain distinct fidelity benefits.

  • The sweep modulates fixed-width windows of eight ranked channels while holding architecture, objective, schedule, and parameter count constant.
  • Ranks 1–8 achieve the strongest aggregate no-reference perceptual performance, although MANIQA peaks at ranks 9–16.MUSIQ, LIQE, CLIP-IQA, and TOPIQ generally increase toward the upper tail.
  • Ranks 33–40 still improve perceptual metrics relative to the unmodulated baseline while achieving the best SSIM and LPIPS in the sweep.Their channels remain 2.3× the per-block average activation magnitude.

D. Detailed Comparison with Existing Methods

Against external diffusion-based SR methods, SPARK-adapted backbones achieve competitive perceptual quality while preserving a broader fidelity–perception balance.

  • The external comparison is intended to position adapted backbones within the broader SR landscape because competing architectures lack massive activations for direct SPARK modulation.
  • 75.35 CLIP-IQA and 4.69 LIQE on RealSR, and 76.32 CLIP-IQA and 4.47 LIQE on DRealSR, surpass every compared method on both metrics.These scores are achieved by TSD-SR + SPARK.
  • TEASR gains the most on perceptual metrics, improving CLIP-IQA by +8.49 and LIQE by +1.01 on DRealSR, and by +7.44 and +0.89 on RealSR.
  • SPARK-adapted models produce sharper, more plausible high-frequency details while preserving global scene structure without noticeable artifacts.

F. Limitations

SPARK’s applicability depends on sparse magnitude-dominant channels and remains evaluated only on DiT-based super-resolution models and three datasets. It reduces adaptation cost but does not accelerate backbone sampling, so inference can remain expensive.

  • SPARK may be less effective in architectures where activation magnitude is distributed more uniformly across channels.The selection procedure relies on sparse magnitude-dominant activation channels, whose strength may vary across architectures.
  • Channel selection is backbone-specific, requiring Phase 1 to be repeated when changing the architecture or checkpoint.
  • The evaluation covers DiT-based super-resolution models and three datasets, leaving convolutional or U-Net backbones, other restoration tasks, and broader degradations open.
  • SPARK reduces adaptation cost but does not accelerate sampling, leaving inference dominated by the frozen SR model, especially for multi-step DiT4SR.
Loading 2609.03813v1…