PaperlayerView exact revision on arXiv

Source-linked AI summary

SHAMISA: SHAped Modeling of Implicit Structural Associations for Self-supervised No-Reference Image Quality Assessment

Mahdi Naseri, Zhou Wang

arXiv:2603.13669v2cs.CVcs.AIcs.LG

TL;DR

NR-IQA needs scalable quality prediction without pristine references or costly human labels. SHAMISA learns distortion-aware and content-sensitive representations from unlabeled distorted images using compositional degradations and dual-source soft relation graphs, then transfers a frozen encoder to quality regression. Across synthetic, authentic, and cross-dataset benchmarks, it reports strong overall performance, including improved robustness and transfer without human quality annotations or contrastive losses.

  • Problem

    NR-IQA requires perceptual quality estimation without pristine references, but supervised learning depends on costly human annotations.

  • Method

    SHAMISA combines continuously parameterized compositional distortions with metadata-driven and structurally intrinsic relation graphs in a non-contrastive self-supervised framework.

  • Results

    SHAMISA achieves strong performance across synthetic, authentic, and cross-dataset NR-IQA benchmarks, with improved robustness and transfer.

  • Takeaways & Limitations

    The framework supports quality prediction from unlabeled distorted images while using a frozen encoder and avoiding human quality annotations and contrastive losses.

Abstract

from arXiv · show

No-Reference Image Quality Assessment (NR-IQA) aims to estimate perceptual quality without access to a reference image of pristine quality. Learning an NR-IQA model faces a fundamental bottleneck: its need for a large number of costly human perceptual labels. We propose SHAMISA, a non-contrastive self-supervised framework that learns from unlabeled distorted images by leveraging explicitly structured relational supervision. Unlike prior methods that impose rigid, binary similarity constraints, SHAMISA introduces implicit structural associations, defined as soft, controllable relations that are both distortion-aware and content-sensitive, inferred from synthetic metadata and intrinsic feature structure. A key innovation is our compositional distortion engine, which generates an uncountable family of degradations from continuous parameter spaces, grouped so that only one distortion factor varies at a time. This enables fine-grained control over representational similarity during training: images with shared distortion patterns are pulled together in the embedding space, while severity variations produce structured, predictable shifts. We integrate these insights via dual-source relation graphs that encode both known degradation profiles and emergent structural affinities to guide the learning process throughout training. A convolutional encoder is trained under this supervision and then frozen for inference, with quality prediction performed by a linear regressor on its features. Extensive experiments on synthetic, authentic, and cross-dataset NR-IQA benchmarks demonstrate that SHAMISA achieves strong overall performance with improved cross-dataset generalization and robustness, all without human quality annotations or contrastive losses.

I. INTRODUCTION

SHAMISA addresses NR-IQA’s costly annotation burden by learning representations from unlabeled distorted images with explicit relational supervision that jointly models content and distortion. Its compositional distortion engine and dual-source graphs provide soft, fine-grained similarity constraints for strong synthetic, authentic, and cross-dataset performance.

  • Motivation: NR-IQA is challenging because perceptual quality depends on complex interactions between distortion and content, while supervised methods require costly human annotations.KADID-10K alone requires over 300,000 subjective ratings.
  • SHAMISA: SHAMISA learns unified content-aware and distortion-aware representations through explicitly constructed relation graphs in a non-contrastive self-supervised framework.Its soft graph relations encode latent perceptual associations more finely than uniform or rigid similarity constraints.
  • Results: SHAMISA achieves strong performance across synthetic, authentic, and cross-dataset NR-IQA benchmarks without human quality labels or contrastive objectives.The encoder is transferred to quality prediction after self-supervised learning.
  • SHAMISA: Dual-source relational supervision combines metadata-driven degradation similarity with structurally intrinsic feature relations, generalizing rigid pairing schemes used by prior SSL-IQA methods.The framework decouples quality learning from fixed class labels and negative sampling.
  • SHAMISA: The compositional distortion engine generates continuously parameterized degradations and groups samples so that one degradation factor varies while other factors remain controlled.This supports structured shifts in representation space and precise metadata generation for graph construction.

III. METHOD

SHAMISA pre-trains an encoder on online-generated distortions using a non-contrastive VICReg-style objective and transfers its frozen representations to NR-IQA through linear regression. Its distortion engine samples continuously parameterized compositions and uses normalized severities to support controlled degradation modeling.

  • Overview and SSL Formulation for NR-IQA: SHAMISA freezes the pre-trained encoder and trains a lightweight linear regressor on its representations for downstream quality prediction.The projector is discarded after self-supervised pre-training, and the backbone is never fine-tuned.
  • Overview and SSL Formulation for NR-IQA: The training batch concatenates pristine reference crops with all generated distorted image sets before computing representations and embeddings.The batch is formed from reference images and Ncomp online-generated compositions.
  • Overview and SSL Formulation for NR-IQA: Human opinion scores are used only after pre-training to fit the regression head, not to learn the encoder representation.This separates self-supervised representation learning from supervised quality-score prediction.
  • Compositional Distortion Engine: A distortion composition applies at most one function from each category, with randomized order and continuously sampled normalized severities.The engine uses seven KADID-10K categories and samples an uncountable family of compositions through continuous severity values.

1) Single-factor variation and trajectories:

SHAMISA constructs composition groups in which exactly one distortion factor varies across severity levels, creating controlled trajectories for relational supervision. These soft graph-weighted relations replace rigid binary invariance while retaining VICReg’s variance and covariance regularization.

  • Single-factor variation and trajectories: Each tiny-batch contains composition groups formed from reference images and a base distortion composition with one selected varying coordinate.The remaining distortion factors are held fixed within the group.
  • Single-factor variation and trajectories: Varying one distortion function while holding others fixed isolates that function’s effect in the learned representation.Each generated severity level applies the resulting composition to the same reference.
  • Explicit Relation Graphs and Graph-weighted VICReg: The framework constructs soft relation graphs whose weighted invariance term brings embeddings together in proportion to relational strength rather than using rigid binary positives.Binary augmentation relations remain a special case when graph weights are restricted to zero or one.
  • Explicit Relation Graphs and Graph-weighted VICReg: SHAMISA remains non-contrastive by preventing collapse with variance and covariance terms instead of negative pairs.Its objective replaces VICReg’s augmentation-paired invariance term with a graph-weighted variant.
  • Explicit Relation Graphs and Graph-weighted VICReg: Explicit soft cross-component edges alleviate the disconnected structure produced by rigid two-view augmentation graphs.The graph is constructed from current representations and then treated with stop-gradient during model updates.

D. Metadata-Driven Graphs

Metadata-driven graphs convert distortion metadata into soft similarity weights that encode both anchor proximity and cross-content similarity. Their monotone severity relationships attract mild or nearby degradations while allowing severe or distant ones to separate.

  • Metadata-Driven Graphs: Metadata-driven weights decrease smoothly with severity from a pristine anchor, making mild degradations closer and strong degradations more distant.The similarity maps are strictly decreasing functions of normalized severity.
  • Metadata-Driven Graphs: Within a distortion composition group, samples with similar severity remain neighbors across different contents.The graph complements content-specific anchor relations with cross-content degradation similarity.
  • Metadata-Driven Graphs: Nearby severity levels receive attraction, whereas distant levels do not, tying similarly degraded images together independently of content.A global Top-Kd sparsifier retains only the strongest matrix entries to control graph density.

3) Reference-reference graph Grr:

The reference-reference graph links pristine images across contents to stabilize a shared high-quality anchor while preserving content diversity. It combines metadata and representation-based relations to shape severity-aware, content-aware embeddings and global prototype organization.

  • Reference-reference graph Grr: Weak links between pristine images across contents stabilize a shared high-quality anchor without forcing content collapse.Variance-covariance regularizers prevent trivial solutions while preserving a coherent pristine neighborhood.
  • Reference-reference graph Grr: The three metadata graphs provide severity-aware and content-aware soft relations for graph-weighted invariance.These relations are aggregated and used to shape quality-sensitive embeddings.
  • Reference-reference graph Grr: Current encoder features define kNN neighborhoods that promote local embedding smoothness with respect to perceptual similarity.The graph retains top similarities per sample after removing self-connections.
  • Reference-reference graph Grr: OT clustering guidance organizes global topology by bringing samples with similar prototype memberships together across contents.Balanced targets enforce approximately uniform cluster marginals, while soft assignments preserve uncertainty during cluster formation.

F. Multi-Source Aggregation with Stop-Gradient

SHAMISA aggregates multiple sparse relation graphs using stop-gradient sources and learned nonnegative weights. A lightweight hypernetwork adapts these weights from graph statistics as the relational sources evolve during training.

  • Multi-Source Aggregation with Stop-Gradient: The aggregation combines Grd, Gdd, Grr, Gk, and Go after applying stop-gradient to each source.The weights are constrained to be nonnegative and sum to one.
  • Multi-Source Aggregation with Stop-Gradient: A lightweight hypernetwork produces graph weights from simple per-graph statistics.The stop-gradient operator is identity in the forward pass and zero gradient in the backward pass.
  • Multi-Source Aggregation with Stop-Gradient: The statistics measure connection mass and sparsity, providing inexpensive scale-agnostic signals for allocating weights without hand-tuning.The hypernetwork aggregates an arbitrary set of sources online as the graphs evolve.
  • Multi-Source Aggregation with Stop-Gradient: Sparse graph weighting evaluates invariance only over retained nonzero edges and regularizes their mass to remain informative.Graph entries are kept in [0, 1] to avoid uncontrolled saturation.

H. End-to-End Optimization and Regression Protocol

After self-supervised pre-training, SHAMISA discards the projector, freezes the encoder, and predicts quality with a linear regressor on encoder features. It is evaluated across synthetic and authentic benchmarks under a frozen-encoder linear-probe protocol, achieving the strongest six-dataset SSL-IQA average.

  • End-to-End Optimization and Regression Protocol: After pre-training, SHAMISA discards the projector, freezes the encoder, and trains a linear regressor on encoder features for human opinion scores.The encoder, projector, prototypes, and aggregation hypernetwork are optimized jointly during pre-training.
  • End-to-End Optimization and Regression Protocol: The evaluation uses frozen features, ridge regression, multiple train/validation/test splits, and SRCC and PLCC on synthetic and in-the-wild datasets.The benchmarks include LIVE, CSIQ, TID2013, KADID-10K, FLIVE, and SPAQ.
  • End-to-End Optimization and Regression Protocol: SHAMISA attains the best SRCC and PLCC on LIVE, CSIQ, and TID2013 while remaining competitive on KADID-10K, FLIVE, and SPAQ.It ranks behind CONTRIQUE on KADID-10K, is second on FLIVE, and narrowly trails Re-IQA on SPAQ.
  • End-to-End Optimization and Regression Protocol: 0.886/0.904 average SRCC/PLCC is SHAMISA’s strongest six-dataset average among the reported SSL methods.The result covers the six-benchmark evaluation under the frozen-encoder linear-probe protocol.
  • End-to-End Optimization and Regression Protocol: The authors attribute the gains to single-factor severity trajectories and dual-source relation graphs within a graph-weighted VICReg objective.They report that the gains do not come from a materially larger training budget.

E. Cross-Dataset Transfer

SHAMISA transfers effectively across synthetic NR-IQA datasets, achieving the best SRCC on most transfer directions without target labels or adaptation. Its strongest gains occur when training data cover fewer distortion types than the target.

  • SHAMISA achieves the best SRCC on 9 of 12 synthetic transfer directions, ranks second on two, and remains competitive on the remaining direction.
  • Relative to ARNIQA, SHAMISA shows its clearest gains when narrower-source datasets transfer to more diverse targets.Examples include LIVE → CSIQ, LIVE → TID2013, CSIQ → TID2013, and TID2013 → KADID-10K.
  • The transfer protocol freezes the encoder, trains a ridge regressor on the source dataset, and evaluates zero-shot on the target test split.Regularization is selected on the source validation split, with identical multi-scale features, five-crop pooling, and median results over 10 reference-disjoint splits.

J. Pre-training Dynamics

Downstream SRCC rises quickly during SSL pre-training and then generally saturates, but the best checkpoint may differ by dataset. Hyperparameter sweeps show broad stability for many settings, with OT alignment especially influential.

  • Pre-training dynamics: SRCC improves rapidly early in pre-training and then saturates across representative NR-IQA benchmarks.The curves show mean performance with ± std error bars over 10 splits.
  • Checkpoint selection: The best checkpoint is dataset-dependent because the SSL objective is not optimized directly for any single benchmark’s SRCC.Some datasets continue improving or remain stable, while others show mild peak-and-decline behavior.
  • Hyperparameter sensitivity: Most capacity and graph-construction hyperparameters exhibit broad SRCC plateaus around the A0 configuration.dh, dz, and kn are typically stable across wide ranges, while increasing K mainly saturates after moderate values.
  • Hyperparameter sensitivity: The OT alignment coefficient η produces the largest cross-dataset SRCC swings, especially on TID2013 and KADID-10K.Weak alignment underuses relational guidance, whereas overly strong alignment can over-constrain representations.
  • Framework overview: SHAMISA combines structured distortion generation with dual-source relational supervision and transfers the learned encoder through a lightweight regressor.The encoder is frozen after pre-training, and the framework is evaluated without human quality labels or contrastive objectives.

APPENDIX A ARCHITECTURES AND OPTIMIZATION

The appendix specifies SHAMISA’s fixed pre-training and evaluation setup, including compositional distortion generation, frozen-encoder transfer, and benchmark protocols. It also describes SHAMISA-FR, which reuses the frozen encoder with a lightweight regressor and remains competitive but trails supervised FR baselines on broader distortion sets.

  • Distortion generation: The compositional engine forms single-factor trajectories by varying one distortion coordinate across five severity levels while holding other coordinates at sampled baselines.The implementation uses Md = 7, B = 2, R = 3, C = 4, and L = 5.
  • Evaluation protocol: Downstream NR-IQA evaluation uses frozen encoder features and ridge regressors across four synthetic and two authentic benchmarks.The appendix documents reference-disjoint synthetic splits, the official FLIVE split, and random 70/10/20 SPAQ splits.
  • Full-reference extension: SHAMISA-FR reuses the frozen no-reference encoder and predicts quality from the absolute feature difference between pristine and distorted images.Only a linear regressor is trained on |href − hdist|, without updating the backbone.
  • Full-reference results: SHAMISA-FR is most competitive among SSL-based FR rows on CSIQ, remains competitive on LIVE, and trails more clearly on TID2013 and KADID-10K.Supervised FR baselines remain stronger because they optimize full networks directly on target datasets.
  • Architectures and optimization: SHAMISA uses one fixed pre-training configuration shared across all datasets, with no dataset-specific tuning during SSL.Table IV summarizes the essential final hyperparameters.

APPENDIX G ADDITIONAL T-SNE VISUALIZATIONS

Additional t-SNE analyses examine SHAMISA representations by severity and subtype on KADID-10K. The embeddings show smooth local severity progressions and clearer separation for blur, compression, and spatial distortions than for color and brightness manipulations.

  • Visualization protocol: The KADID-10K t-SNE uses 10,125 image-level encoder representations extracted from multiple crops and reduced to 50 PCA dimensions first.The visualization uses fixed random seeds and specified scikit-learn t-SNE settings.
  • Severity visualizations: Severity-colored embeddings show smooth local severity progressions for several coarse distortion families, although visually related distortions still overlap.Colors encode severity levels 1 through 5.
  • Subtype visualizations: Blur, compression, and spatial distortions show the clearest subtype-level separation in SHAMISA’s embeddings.The subtype visualization focuses on six KADID-10K distortion families.
  • Subtype visualizations: Color and brightness-related manipulations remain more entangled than blur, compression, and spatial distortion subtypes.The comparison identifies where representation structure is less cleanly separated.

APPENDIX J ADDITIONAL ABLATIONS

Ablations identify OT-guided global structure as SHAMISA’s dominant structural prior, while metadata and intrinsic structural graphs provide complementary supervision. Additional diagnostics show early feature stabilization and quantify sensitivity across six NR-IQA datasets.

  • Additional ablations: Removing OT alignment causes the largest ablation drop, reducing SRCCg to 0.7887.Removing the OT-derived global graph causes the strongest single-source removal drop, to 0.8600.
  • Additional ablations: Severity-dependent edge weighting is helpful but secondary, with F1 reducing SRCCg to 0.8736 compared with the larger OT-alignment loss.The complete catalog includes finer-grained graph removals, OT sparsification, and an additional distortion-engine setting.
  • Training diagnostics: Feature correlation proxies for H and Z drop rapidly before stabilizing, while H’s effective dimensionality and invariance proxy also settle after early training.The reported trends are consistent with reduced redundancy and progressively closer paired views.
  • Sensitivity analysis: One-at-a-time sweeps measure hyperparameter sensitivity as the largest-minus-smallest SRCC across LIVE, CSIQ, TID2013, KADID-10K, FLIVE, and SPAQ.Larger Δ indicates higher sensitivity.
Loading 2603.13669v2…