PaperlayerView exact revision on arXiv

Source-linked AI summary

Multidimensional Observer Model and Perceptual Dimensions of Human Image Quality Assessment

Sheng Zhao, Weikai Lin, Yuhao Zhu

arXiv:2609.38487v1cs.CV

TL;DR

Human image-quality judgments matter for everyday and machine-vision tasks, but existing metrics are designed mainly for prediction rather than probing perceptual structure. This paper models images as distributions in a multidimensional latent space and fits the model to human judgments. It finds that the perceptual space is low-dimensional and task-dependent.

  • Problem

    Existing image-quality metrics primarily optimize prediction, leaving open how perceptual space organizes images, how judgments form, and how its dimensionality and structure vary across tasks.

    Source: §1 · p. 1

  • Method

    A multidimensional observer model represents images as noisy distributions in a latent space constrained by primate visual representations and is fitted to behavioral choice data.

    Source: §1 · pp. 1–2

  • Results

    Human image-quality judgments are accounted for by extremely low-dimensional perceptual spaces, with low-level tasks generally saturating within 10 dimensions and high-level tasks requiring about an order of magnitude more.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The perceptual space for image-quality judgment is task-dependent despite shared retinal representations across tasks.

    Source: §1 · p. 2, §7 · p. 10

    Scaling each image’s mean and covariance proportionally leaves triplet choice probabilities unchanged, making distributions difficult to compare across images not in the same triplet.

    Source: §6 · p. 9

Abstract

from arXiv · show

Judging image quality is not only ecologically relevant to everyday human tasks, but also underpins many machine vision tasks such as image generation. This paper proposes a framework to understand the inherent perceptual space underlying image quality judgment in humans. We propose a multi-dimensional observer model that represents images as distributions in a latent perceptual space and that models human judgment as comparing noisy samples. Being constrained by neural representations in the primate ventral stream and fit to large-scale behavioral data, the model enables analysis of perceptual structure while matching the predictive power of existing metrics. Using this model, we find that the perceptual spaces needed to account for image quality judgment in humans are extremely low-dimensional compared to the image space even when considering its sparsity. The exact structure of the space (e.g., dimensionalities, information encoded) varies between low-level and high-level quality judgments, suggesting that, despite a shared retinal encoding in the beginning, humans selectively construct task-dependent perceptual spaces in visual decision making.

1 Introduction

The paper introduces a multidimensional observer model to probe the perceptual structure underlying human image-quality judgments. It finds that the required perceptual spaces are low-dimensional and vary with task complexity.

1 Introduction

  • The observer model represents each image as a noisy distribution in a multidimensional latent space and models quality through comparisons of noisy samples.It formalizes the assumptions that image representations vary within and across observers and that greater distance from the clean reference indicates lower quality.Source: §1 · pp. 1–2
  • The model approaches human performance and is on par with or better than existing perceptual metrics across BAPPS, PieAPP, and NIGHTS.It is constrained to align with representations in the primate retinal and ventral stream and fitted to behavioral data from 2AFC experiments.Source: §1 · p. 2
  • Human image-quality judgments can be accounted for by extremely low-dimensional perceptual spaces, with saturation generally within 10 dimensions for low-level tasks and about an order of magnitude higher for high-level tasks.The perceptual dimensionality is much lower than the image-space dimensionality, even considering natural-image sparsity.Source: §1 · p. 2
  • Low-level judgments gain only slightly from V1/V2-to-V4 information, and adding IT signals degrades performance; high-level judgments benefit from high-level cortical information, with IT markedly outperforming V1.This contrast supports task-dependent perceptual spaces despite shared retinal representations.Source: §1 · p. 2

2 Related work

Prior perceptual metrics primarily target predictive accuracy, while this work focuses on recovering the dimensionality and structure of the perceptual space. Related psychometric approaches often use one-dimensional scales, and recent multidimensional studies address other similarity tasks.

2 Related work

  • Existing perceptual metrics improve prediction through feature encoders, multiscale aggregation, or task-specific calibration, but do not aim to recover perceptual-space dimensionality and structure.The cited examples span classic measures, learning-based metrics, and psychophysical visible-difference predictors.Source: §2 · p. 2
  • Psychometric scaling methods, including classical laws and probabilistic models, typically map stimuli onto a one-dimensional scale, limiting their ability to capture underlying perceptual-space structure.The paper focuses on that structure in image-quality judgment.Source: §2 · p. 2
  • Recent studies examine perceptual-space dimensionality for object and material similarity, whereas this work studies image quality; those approaches build on Bradley–Terry.The paper discusses the connection between Bradley–Terry and its observer model in Section 6.Source: §2 · p. 2

3 Method

The observer model treats image responses as noisy distributions and predicts pairwise quality choices by comparing sampled distances to a reference. It is fitted to human choice data using a flexible, biologically grounded representation and likelihood optimization.

3.1 Observer Model

  • In triplet 2AFC, the observer samples a perceptual response for the reference and each alternative, then chooses the alternative closer to the reference.Noise allows responses to vary across trials and observers; the model represents each image with a Gaussian whose covariance can capture image-specific ambiguity.Source: §3.1 · pp. 3–4
  • The model predicts choice probability from the expected comparison of sampled distances, using a weighted Minkowski distance whose exponent controls whether dimensions act separably or integrally.The dimension weights compensate for differences in natural channel magnitudes, and the model also extends to other psychophysical paradigms.Source: §3.1 · p. 4

3.2 Fitting the Observer Model Using Behavioral Data

  • To allow unknown low dimensionality while preserving smooth stimulus responses, the model projects biologically grounded image features into a tunable-dimensional latent mean representation.A linear projection provides flexible control of N while preserving the feature dimensions up to an affine transformation.Source: §3.2 · p. 4
  • The image-dependent covariance is parameterized through a Cholesky factor, with a lower-triangular factor and nonzero diagonal ensuring positive semidefiniteness.A small MLP generates the factor, and softplus ensures positive diagonal elements.Source: §3.2 · pp. 4–5
  • The model is fitted by maximizing the likelihood of human 2AFC choices, approximating expected choice probabilities with Monte Carlo samples and using gradient estimators to optimize them.The method uses the straight-through estimator for the indicator and the reparameterization trick for differentiable samples.Source: §3.2 · p. 5

4 Experiment Setup

The experiments evaluate the observer model on triplet-choice and rating datasets using visual feature encoders and established comparison measures. They sweep latent dimensionality to estimate when predictive performance saturates.

4 Experiment Setup

  • The experiments use CORnet-S features aligned with V1, V2, V4, and IT, plus contrast-pyramid and pretrained VGG-16 encoder alternatives.CORnet-S outputs are pooled by area, with a concatenated 960-dimensional representation called CORnet-S Multi; encoder features are not fine-tuned.Source: §4 · p. 5
  • The evaluation spans BAPPS and PieAPP low-level distortion triplets, NIGHTS text-to-image variation, and four rating-based IQA datasets.The additional datasets are LIVE, KADID-10k, CSIQ, and TID2013.Source: §4 · pp. 5–6
  • For BAPPS and PieAPP, the study reports human-model agreement and choice-distribution KL divergence; for NIGHTS, it reports 1 minus binary error rate.Theoretical ceilings and inter-observer agreement contextualize the agreement measure.Source: §4 · p. 6
  • The protocol sweeps latent dimension N and defines n* as the smallest dimension explaining 95% of the model’s above-chance performance.Optimization uses Adam with a 10^-3 learning rate, cosine annealing, batch size 256, 50 epochs, and 64 Monte Carlo samples.Source: §4 · pp. 6–7

5 Results

The observer model predicts human 2AFC judgments competitively while revealing low-dimensional, task-dependent perceptual spaces. The dimensions and visual features that best account for judgments vary with task complexity.

5.1 Our Observer Model Predict Human 2AFC Tasks Accurately

  • 0.688 agreement on BAPPS beat all baselines; on PieAPP the model narrowly trailed the best method, and on NIGHTS it tied DISTS behind DreamSim.On KL divergence, it performed best against CVVDP and PieAPP on BAPPS and ranked second on PieAPP.Source: §5.1 · p. 7

5.2 Extremely Low-Dimensional Perceptual Spaces Account for Human Judgment

  • At matched sparsity, the DCT control scored about 0.1 lower in agreement; matching the observer’s N=6 performance required roughly two orders of magnitude more DCT dimensions.The control tests whether low saturating dimensionality follows simply from natural-image frequency-domain compressibility.Source: §5.2 · pp. 7–8
  • n*=3–8 on BAPPS and PieAPP, versus 96 at IT on NIGHTS, shows that saturating dimensionality increases for higher-level judgments.Across encoder choices, low-level tasks generally saturate around five dimensions, while high-level tasks need about 10× more; the observer model’s performance plateaus within a handful of dimensions on BAPPS and PieAPP, but requires more on NIGHTS.Source: §5.2 · pp. 7–8, §5.3 · p. 8

5.3 Task Complexity Shapes Perceptual Dimensions and Representational Requirements

  • Low-level judgments align best with early-to-intermediate visual areas, whereas high-level judgments benefit from IT; the contrast-pyramid representation performed worst across both tasks.Moving to IT reduced alignment on low-level judgments, while V1 and V2 performed markedly worse than later representations on high-level judgments.Source: §5.3 · p. 8
  • A BAPPS-trained model transferred to PieAPP with only a negligible accuracy drop, but degraded substantially on NIGHTS compared with a NIGHTS-trained model.This transfer pattern corroborates that perceptual-space structure varies with task complexity.Source: §5.3 · p. 8

5.4 Interpreting the Perceptual Dimensions

  • In the learned BAPPS N=2 space, z1 encodes texture or spatial frequency and z2 encodes color; higher-dimensional spaces yield finer-grained, harder-to-name dimensions.The covariance field varies smoothly, and dimensions for high-level tasks encode more object semantics while low-level dimensions encode more color and texture information.Source: §5.4 · p. 9, §5.3 · p. 9

6 Discussion

The observer model has similar predictive power to the Bradley–Terry model and can provide a lower-level interpretation of it. A limitation is that its triplet choice probabilities do not fix a shared scale for comparing distributions across different triplets.

6 Discussion

  • The Bradley–Terry model and this method have similar predictive power, while the observer model offers a low-level interpretation of the phenomenological model.Bradley–Terry is a classical psychometric scaling approach for predicting pairwise choice distributions and assumes image quality follows a Gumbel distribution.Source: §6 · p. 9
  • Proportional scaling of each image’s mean and covariance leaves triplet choice probabilities unchanged, making distributions difficult to compare across different triplets.The authors suggest gathering direct observations of the mean field, for example through neural recordings during IQA tasks or cross-content judgments.Source: §6 · pp. 9–10

7 Conclusion

The paper concludes that human image-quality judgments can be modeled in an extremely low-dimensional, task-dependent perceptual space, providing a bridge between behavioral data and learned perceptual representations.

7 Conclusion

  • Human image-quality judgments can be accounted for by an extremely low-dimensional perceptual space whose structure varies with task.The authors frame the framework as a way to probe human perceptual processing of images by connecting behavioral data with the learned perceptual space.Source: §7 · p. 10

A Architecture and Methodology Details

The supplementary methods describe frozen feature encoders paired with a trainable observer head, along with sampling, dataset, and control details. A separate retrieval analysis illustrates dimensions learned for PieAPP and NIGHTS.

A Architecture and Methodology Details

  • The observer pipeline uses cached features from frozen CORnet-S, VGG, or contrast-pyramid encoders before a trainable head; CORnet-S features align with ventral-stream areas.The encoder variants include pooled and spatial-grid VGG features, while the CORnet-S multi-area representation concatenates pooled V1, V2, V4, and IT features.Source: A Architecture and Methodology Details · p. 14
  • The covariance field is parameterized through a Cholesky factor, and triplet choice probabilities are estimated by repeatedly sampling neural responses for the reference and two candidates.The Cholesky diagonal is constrained to be positive; sampling uses reparameterization, while the authors choose a straight-through estimator over a sigmoid alternative because the latter makes the covariance field harder to interpret.Source: A Architecture and Methodology Details · p. 14
  • At N=3 and N=12 on BAPPS, the trainable head is one to four orders of magnitude smaller than other learnable metrics and never exceeds 25K parameters.Table 4 compares trainable, frozen, and total parameter counts.Source: A Architecture and Methodology Details · p. 14, Table 4 · p. 15
  • The DCT control tests whether frequency-domain image sparsity alone accounts for the observer’s low saturating dimensionality by matching coefficient and perceptual-dimension budgets.It retains top-K DCT coefficients per YUV channel, reconstructs images, and feeds them to the observer model.Source: A Architecture and Methodology Details · p. 15
  • The evaluation includes three triplet 2AFC datasets—BAPPS, NIGHTS, and PieAPP—and four rating-based IQA datasets, used as released without filtering or relabeling.The triplet datasets cover low-level distortions and high-level synthesized images; the rating-based datasets provide MOS or DMOS scores.Source: A Architecture and Methodology Details · pp. 15–16

B.1 Perceptual Space Interpretation at N=2 on PieAPP and NIGHTS

  • At N=2, PieAPP retrieval varies along texture-like and color-like axes, while NIGHTS retrieval varies along scene multiplicity and color axes.For NIGHTS, one axis contrasts single foreground subjects with multiple repeating elements; the other ranges from more colored to near-monochromatic images.Source: §B.1 · p. 16

B.2 Perceptual Space Interpretation at High Dimensionalities

The interpretation analysis links learned perceptual dimensions to visual content across the ventral-stream hierarchy, with dataset-dependent semantic levels. The dimensions are also largely independent.

B.2 Perceptual Space Interpretation at High Dimensionalities

  • CORnet-S channels progress from color and low-level texture in V1, through texture in V2 and parts in V4, to whole objects in IT.The ventral-stream level recruited by the perceptual space tracks the dataset’s semantic level.Source: §B.2 · pp. 16–17
  • The N=2 visualizations show a texture/structure direction crossed with a color direction for PieAPP or a scene-content direction for NIGHTS.Each grid cell represents an image near that point in the learned space; traversing an axis varies one latent dimension from low to high.Source: §B.2 · p. 17
  • BAPPS and PieAPP dimensions remain dominated by color and texture, while NIGHTS dimensions draw substantially on IT and reach part- and object-level semantics.At higher dimensionalities, concepts within each dimension become more diverse and mixed.Source: §B.2 · p. 17

B.3 Dimensions are Independent

  • Across the datasets and dimensionalities examined, learned dimensions are largely independent, with average off-diagonal correlation below 0.25.Figure 7 reports pairwise Pearson correlations on the validation set for BAPPS, PieAPP, and NIGHTS at N = 2, 4, 8, and 16.Source: §B.3 · pp. 18–19

C Results on Different Feature Encoders and Projection Choices

Across evaluated encoders, the low-dimensional saturation conclusion holds under learned projections, while PCA performs worse than learned projection. Table 5 reports peak performance and saturation dimensionality across three benchmarks and both projection choices.

C Results on Different Feature Encoders and Projection Choices

  • Table 5 compares learned projection and PCA across BAPPS, NIGHTS, and PieAPP, reporting peak performance alongside saturating n∗.BAPPS reports agreement/n∗ and KL/n∗, NIGHTS accuracy/n∗, and PieAPP agreement/n∗.Source: Table 5 · p. 19, C Results on Different Feature… · p. 19
  • Under learned projection, BAPPS saturates at n∗ = 3–7 under agreement and similarly low dimensionality under KL; PCA performs worse across encoders.The saturation diagnosis does not depend on the choice of agreement or KL metric.Source: Table 5 · p. 19, C Results on Different Feature… · p. 20

D Results on Cross-Dataset Prediction

Cross-dataset transfer from BAPPS nearly reaches the in-domain ceiling on PieAPP but remains below it on NIGHTS. The residual NIGHTS gap persists as dimensionality and visual hierarchy increase.

D Results on Cross-Dataset Prediction

  • BAPPS-trained observers transfer to PieAPP with almost no loss for higher-level CORnet-S encoders, but remain several points below the NIGHTS in-domain ceiling across encoders.For NIGHTS, the residual gap does not close as N grows or as the encoder moves up the visual hierarchy.Source: D Results on Cross-Dataset Prediction · p. 20
  • Table 6 reports cross-domain peaks and saturation points for B→N accuracy, N→B agreement, and B→P agreement under the learned projection.The target-dataset saturation dimensionality is often higher than its in-domain counterpart.Source: D Results on Cross-Dataset Prediction · p. 20, Table 6 · p. 20

E Extension to Rating-Based IQA Datasets

The observer model extends to rating-based IQA by using expected perceptual distance as a score, and it performs strongly across the evaluated benchmarks. The appendix also compares it with Bradley–Terry and describes MAP fitting with a Wishart prior.

E Extension to Rating-Based IQA Datasets

  • Expected Minkowski distance between an image distribution and the reference distribution can serve as the image’s perceptual score for rating-based IQA.The model is evaluated zero-shot on TID2013, CSIQ, and LIVE Release 2; KADID-10k is excluded because the model is trained on its full MOS labels.Source: E Extension to Rating-Based IQA Datasets · p. 21
  • Across six datasets, the observer model achieves a best average rank of 2.83 and ranks in the top three on five.On the three rating-based benchmarks, it often ranks among the top three across all three reported metrics.Source: E Extension to Rating-Based IQA Datasets · p. 21

F Comparison with Bradley–Terry Model

  • The Bradley–Terry and full observer models have similar predictive power, while the physiologically grounded observer model permits examining perceptual-space dimensions.The original Bradley–Terry model cannot examine those dimensions.Source: F Comparison with Bradley–Terry Model · p. 23

G MAP Using Wishart Prior on the Covariance

  • MAP fitting adds a prior on the covariance matrix field to the likelihood and estimates parameters by maximizing their combined objective.The paper uses a weak Wishart prior and integrates it with the likelihood through MAP estimation.Source: G MAP Using Wishart Prior on the… · p. 23

G.1 MAP with Wishart Distribution Prior

  • The Wishart prior uses ν = n + 2 and V = 1/ν I, giving an isotropic prior with mean E[Σ] = I.The degrees of freedom and scale matrix could in theory be learned, but the authors leave that for future work.Source: §G.1 · p. 24

G.2 Results

The Wishart prior produced no consistent predictive gain, while fitted Minkowski parameters differed across tasks and channel weights were nearly uniform.

G.2 Results

  • In-domain MAP slightly reduced performance (−0.3% BAPPS agreement; NIGHTS ∆=−0.1% at λ=0.05 and −0.5% at λ=1.0), while cross-domain effects were roughly neutral.The tests compare MAP with maximum likelihood; cross-domain evaluations transfer models between BAPPS and NIGHTS.Source: §G.2 · p. 24, Table 11 · p. 24, Table 12 · p. 25
  • The dimensionality finding remained unchanged; the authors suggest MLE already learns covariance effects that the Wishart prior might capture.The baseline jointly learns the mean field and covariance matrix.Source: §G.2 · p. 24

H Parameters of Weighted Minkwoski Distance Across the Visual Hierarchy

  • BAPPS p rises from near 1 at V1 to near 2 at IT, versus NIGHTS’ 1.3–1.7; lower p indicates more separable low-level than integral high-level dimensions.The paper relates p≈1 to separable dimensions and p≈2 to integral dimensions.Source: H Parameters of Weighted Minkwoski… · pp. 24–25
  • At saturation, channel weights were nearly uniform: variation was 5–17% on BAPPS (V1: 38% at n*=4) and at most 6% on NIGHTS, with no channel dominating.The paper notes that differences across dimensions could potentially be learned through the projection matrix.Source: H Parameters of Weighted Minkwoski… · p. 25

Related papers

Loading 2609.38487v1…