PaperlayerView exact revision on arXiv

Source-linked AI summary

Blind Stereoscopic Omnidirectional Image Quality Assessment Using Predictive Coding Hierarchy

Wei Zhou, André Kaup

arXiv:2608.28798v1cs.CV

TL;DR

SOI quality assessment is challenging because users browse local viewports while binocular vision and global scene integration shape perceived quality. PCH models this process with local cyclopean perception, global predictive coding, and a quality regressor, achieving competitive and consistently improved performance against representative state-of-the-art methods. Ablations support complementary contributions from saliency-aware local statistics and predictive coding modelling, while experiments also verify generalization beyond SOI scenarios.

  • Problem

    SOI quality assessment must account for freely browsed viewports, binocular mechanisms, and integration of local perceptual evidence into a global scene representation.

  • Method

    PCH combines viewport-based cyclopean and saliency modelling with binocular luminance fusion, predictive coding hierarchy modelling, and regression of local-global perceptual features.

  • Results

    PCH achieves competitive and consistently improved performance compared with representative state-of-the-art quality assessment methods.

  • Takeaways & Limitations

    Ablations support complementary roles for saliency-aware local statistics and predictive coding modelling, while experiments verify generalization beyond SOI scenarios.

Abstract

from arXiv · show

Stereoscopic omnidirectional images (SOIs) have provided users with newly immersive quality of experience in virtual reality environments. However, developing efficient and accurate perceptual quality assessment metrics for SOIs remains challenging due to many factors such as freely changeable field of views and binocular vision. In this paper, based on the characteristics of the human visual system (HVS), we propose a Predictive Coding Hierarchy-inspired metric (PCH) for blind/no-reference stereoscopic omnidirectional image quality assessment. Motivated by the viewing process of SOIs, the proposed PCH includes a local cyclopean perception module, a global predictive perception module, and a visual quality regressor. First, observers browse different spherical sceneries from viewports, and aggregate the local visual information to infer the perceptual quality of SOIs. Therefore, we extract various viewports, followed by cyclopean conversion and saliency detection to approach the perception and attention of the human brain. After the local aggregation, viewers then infer the global scene in their minds. Based on the binocular mechanism, we fuse left and right views to perform predictive coding hierarchy modelling. Finally, the visual quality regressor is exploited to obtain the ultimate quality score related to both local and global perceptual cues. Extensive experiments demonstrate that the proposed PCH achieves competitive and consistently improved performance compared with state-of-the-art quality assessment methods.

I. INTRODUCTION

SOI quality assessment must account for viewport-based browsing, binocular perception, and the integration of local evidence into a global scene representation. PCH addresses these challenges with local cyclopean and global predictive perception modules for blind assessment.

  • SOIs are difficult to assess because users perceive changeable local viewports, binocular mechanisms influence quality, and observers integrate local evidence into a global representation.
  • PCH simulates local perception by extracting multiple viewports, performing cyclopean conversion, and applying saliency detection to characterize attention-driven quality cues.
  • PCH models global scene perception by fusing left and right views into a luminance map and applying predictive coding hierarchy modelling.
  • The framework combines local and global perceptual features through a visual quality regressor to predict the final blind quality score.
  • Existing omnidirectional IQA methods use viewport-based or spherical-domain approaches, but most do not explicitly model binocular perception in stereoscopic omnidirectional images.

B. Stereoscopic Image Quality Assessment

Prior stereoscopic and blind IQA methods provide relevant components, but SOIs require an explicit model of viewport browsing, binocular perception, and global scene integration. PCH combines these elements through unified local-global predictive perception.

  • Stereoscopic IQA must account for binocular fusion, rivalry, disparity perception, and depth-related discomfort beyond conventional 2D image quality assessment.
  • Planar stereoscopic IQA methods cannot be directly applied to SOIs because SOIs add omnidirectional browsing, spherical distortions, and cross-viewport scene integration.
  • Existing blind SOI methods lack explicit modelling of how local browsing evidence is integrated into a global scene representation.
  • Predictive coding represents perception as hierarchical inference in which top-down predictions and bottom-up prediction errors are iteratively exchanged.
  • PCH combines local viewport perception with global predictive coding hierarchy modelling and adds cyclopean, saliency-aware, and binocular luminance mechanisms.
  • Unlike SOIQE, PCH is blind/no-reference and models hierarchical perception directly from distorted SOI content rather than using pristine references.

A. Overview of the Proposed Framework

PCH extracts multiple viewport pairs, models local binocular perception through cyclopean fusion and saliency, then captures global cues with fused luminance and predictive coding. The resulting local and global descriptors support blind quality prediction.

  • 1) Viewport Extraction:: The PCH pipeline extracts multiple viewport pairs to simulate how users browse stereoscopic omnidirectional content.The default configuration uses N = 6 viewports as a trade-off between prediction accuracy and computational efficiency.
  • The global module constructs a fused luminance representation from binocular views and applies predictive coding hierarchy modelling to capture holistic perceptual regularities.
  • The final blind quality score is predicted from descriptors representing both local perceptual cues and global distortion-related abnormalities.
  • 2) Cyclopean Perception Modelling:: The local cyclopean module approximates binocular fusion by estimating disparity, warping the right viewport, and synthesizing an information-guided cyclopean representation.
  • Saliency-aware local statistics characterize attention strength, spatial dispersion, and structural complexity in perceptually salient regions.

3) Saliency Detection and Attention Modelling:

The saliency module approximates visual attention by generating saliency maps from downsampled cyclopean viewports, emphasizing distinctive regions for local quality perception.

  • Saliency detection approximates visual attention because visually distinctive regions receive greater focus than non-salient areas.
  • Cyclopean viewports are downsampled using low-pass filtering and uniform subsampling before saliency computation.A denotes the low-pass filter and convolution produces the downsampled representation.
  • The downsampled viewport is transformed into the frequency domain with the discrete cosine transform, retaining only coefficient signs.
  • An inverse DCT followed by squaring produces a saliency map that highlights visually distinctive regions for local quality perception.

4) Local Feature Extraction and Aggregation:

Local features summarize attention patterns in each saliency map and concatenate descriptors across viewports to represent local perceptual quality.

  • Each saliency map captures the spatial distribution of visual attention and highlights perceptually dominant regions.
  • Three statistics describe each saliency map: mean saliency strength, standard deviation of attention dispersion, and entropy of salient-region complexity.Entropy is computed from 8-bit quantized saliency values for stable histogram estimation.
  • These statistics characterize attention strength, dispersion, and structural complexity, which distortions can alter through changes in salient-region organization.
  • When multiple viewports are used, their saliency-based descriptors are concatenated to form the aggregated local representation.

C. Global Predictive Perception Module

The global predictive perception module constructs a fused luminance stimulus and processes it across multiple spatial scales with HVS-inspired nonlinear filtering.

  • The module models global scene perception as integration of local evidence through predictive-coding-inspired interactions between feedforward errors and feedback predictions.
  • 1) Binocular Luminance Fusion: Left and right omnidirectional views are converted to luminance and fused by absolute summation to form a global perceptual stimulus.The fused map enhances binocular-consistent structures and distortion-sensitive variations.
  • 2) Multi-scale Preprocessing: The fused luminance map is processed at multiple resolutions, motivated by hierarchical receptive-field properties of the human visual system.Four scales are used: sk ∈ {1, 2, 4, 8}.
  • 2) Multi-scale Preprocessing: At each scale, a Laplacian-of-Gaussian filter followed by a hyperbolic tangent nonlinearity models center-surround responses and suppresses extreme activations.

3) Block-wise Predictive Coding Inference:

Block-wise predictive coding operates on multi-scale response maps, iteratively refining latent coefficients and aggregating coding outputs into global descriptors for quality prediction.

  • Unlike the cited full-reference metric, PCH performs block-wise predictive coding on binocularly fused, multi-scale omnidirectional luminance maps for distortion characterization.
  • Each scale partitions the preprocessed response map into non-overlapping 16 × 16 blocks, which are vectorized for coding.
  • A fixed offline-learned dictionary represents block inputs through latent coefficients, producing top-down predictions and feedforward prediction errors.
  • Latent coefficients are iteratively updated through feedback refinement using fixed parameters and a gating term tied to the hyperbolic-tangent derivative.
  • After optimization, coding coefficients and residual errors are averaged across blocks and concatenated across scales into global descriptors.The resulting representation captures top-down coding responses and bottom-up residual irregularities.

D. Visual Quality Regressor

PCH concatenates local saliency-aware cyclopean features with global predictive-coding representations, then uses ε-SVR with an RBF kernel to predict subjective quality scores.

  • Feature Fusion: PCH concatenates local cyclopean descriptors and global predictive-coding features into a fused feature vector.Local features use saliency-aware viewport statistics, while global features contain predictive-coding coefficients and prediction errors.
  • Feature Construction: The local representation summarizes saliency maps using mean, standard deviation, and entropy, while the global representation averages predictive-coding quantities across four scales.These descriptors capture saliency strength, attention dispersion, structural complexity, and predictive-coding irregularities.
  • SVR-based Quality Prediction: An ε-support vector regression model with an RBF kernel maps the fused features to the predicted quality score.The regressor is selected for nonlinear fitting capability and robustness with relatively limited IQA training data.

IV. EXPERIMENTAL RESULTS AND ANALYSIS

The experiments evaluate PCH across stereoscopic omnidirectional and conventional stereoscopic databases using repeated train-test splits and standard correlation and error metrics. On the primary benchmark, PCH achieves the strongest reported rank and linear correlations among the compared methods.

  • Experimental Settings: Experiments use three public databases, with SOLID as the primary stereoscopic omnidirectional benchmark and LIVE Phase I and II for conventional stereoscopic content.SOLID contains distorted SOIs, while the LIVE databases assess robustness across other stereoscopic modalities.
  • Datasets: The SOLID database contains 276 distorted SOIs with JPEG or BPG compression, including 84 symmetric and 192 asymmetric distortions.Each image is associated with a Mean Opinion Score collected through standardized subjective testing.
  • Datasets: The LIVE Phase I database contains 20 pristine stereoscopic pairs and 365 symmetrically distorted images spanning five distortion types.Its subjective quality labels are provided as differential mean opinion scores.
  • Datasets: The LIVE Phase II database contains 8 original pairs and 360 distorted pairs, including 120 symmetric and 240 asymmetric cases.Each distorted pair has an associated differential mean opinion score.
  • Evaluation Protocol: Evaluation reports SRCC, PLCC, and RMSE over 100 random 80%-20% train-test splits, with higher correlations and lower RMSE indicating better performance.PLCC and RMSE are computed after nonlinear regression following the VQEG protocol.
  • Overall Performance: 0.938 SRCC and 0.947 PLCC are achieved by PCH on the reported overall comparison, surpassing the strongest competing NR OIQA model BPGI.PCH also outperforms existing NR OIQA methods and surpasses FR metrics in the reported comparison.

C. Performance Under Different Distortion Conditions

PCH maintains strong performance across symmetric and asymmetric distortions, compression schemes, module ablations, viewport settings, and conventional stereoscopic databases. The results also identify saliency modelling and joint local-global representation as important design choices.

  • Different Distortion Conditions: PCH achieves consistently superior correlations across symmetric and asymmetric distortions and maintains the best prediction accuracy under JPEG and BPG compression.The results specifically indicate robustness to binocular quality imbalance and asymmetric degradation between views.
  • Ablation Study: The local cyclopean module alone achieves SRCC = 0.917, while the global predictive-coding module achieves SRCC = 0.908.These results show that both saliency-aware viewport perception and holistic irregularity modelling provide competitive quality cues.
  • Ablation Study: Combining the local and global modules improves prediction accuracy, with more than 2% SRCC gain over the local-only variant.The reported gain indicates that predictive-coding modelling captures global distortion irregularities not explained by local saliency statistics alone.
  • Viewport Number and Saliency: Removing saliency consistently degrades performance, whereas increasing viewports from 6 to 20 provides only marginal improvement at higher computational complexity.PCH therefore adopts 6 viewports as the default efficiency-accuracy trade-off.
  • Conventional Stereoscopic IQA: On LIVE Phase I and Phase II, PCH achieves the best correlation with subjective scores among the compared stereoscopic IQA methods.The comparisons include existing stereoscopic IQA baselines and the representative SOI metric SOIQE.
  • Patch-Size Analysis: Using 128×128 patches yields the best SRCC and PLCC on both LIVE databases, although a smaller patch slightly reduces RMSE on LIVE Phase I.The medium-scale setting is selected for overall correlation accuracy and stability.

G. Discussion

The discussion reports that PCH models quality through complementary local attention and global predictive inference, with experiments supporting its effectiveness and broader applicability. Future work targets more efficient learning and richer temporal, depth, discomfort, and attention modelling.

  • PCH explicitly models perceptual inference from local attention allocation to global predictive refinement.
  • Saliency-aware local statistics capture distortion-sensitive attention, while predictive coding characterizes global structural irregularities through error-driven inference.
  • The complementary local and predictive mechanisms explain the performance gain observed when both modules are integrated.
  • Experiments on LIVE stereoscopic IQA databases indicate that the framework generalizes beyond stereoscopic omnidirectional image scenarios.
  • Extensive experiments show competitive and consistently improved performance over representative state-of-the-art methods, with component ablations validating effectiveness and complementarity.
  • Future work will extend the framework to video-based omnidirectional assessment and investigate temporal dynamics, depth, visual discomfort, temporal attention, and adaptive viewport sampling.
Loading 2608.28798v1…