Source-linked AI summary
MetaRefine: Score Refinement with Multi-Level Features for Image Assessment
Chuanzhi Xu, Chengkun Yue, Haolan Guo, Xuanhua Yin, Boyu Wei, Zihan Deng, Qiang Qu, Weidong Cai
TL;DR
Preprocessing can obscure original-image geometry and composition, while existing assessment models contain additional multi-level visual evidence. MetaRefine uses that geometry and evidence with frozen model scores to refine predictions, improving metrics across 100 configurations on 21 datasets with low median parameter overhead.
Problem
Preprocessing can obscure original-image geometry and composition, while existing assessment models contain multi-level visual evidence beyond their final scores.
Source: §1 · pp. 1–2Method
Results
Across 100 configurations on 21 datasets, average gains were 7.55% PLCC and 7.10% SRCC; MAE and RMSE fell 23.27% and 21.93%, with 1.11% median parameter overhead.
Source: §5 · p. 8Takeaways & Limitations
MetaRefine preserves the original scoring pipeline, trains only residual heads, and further improves state-of-the-art IQA and IAA models.
Source: §5 · p. 8
Abstract
from arXiv · showhide
An image's original dimensions, aspect ratio, and composition provide important cues that can affect its technical and aesthetic quality. Image assessment methods use resizing, cropping, padding, or patch sampling to accommodate different input sizes. However, these operations can alter or obscure original geometric and compositional cues. Existing assessment models also encode rich multi-level visual evidence that can be further extracted and exploited to improve existing models. This motivates us to combine visual evidence with original-image geometry for score refinement without changing the original scoring pipelines. In this paper, we introduce MetaRefine, a metadata-aware score refinement architecture whose lightweight residual heads predict score corrections from geometry, multi-level visual features, and the frozen model's original score. Across 100 configurations on 21 datasets, MetaRefine improves PLCC in 88 configurations and SRCC in 85, with average relative gains of 7.55% and 7.10%, respectively. It reduces MAE in 90 configurations and RMSE in 89, with average relative reductions of 23.27% and 21.93%, respectively. The median parameter overhead is only 1.11%. MetaRefine can further improve state-of-the-art (SOTA) assessment models on multiple benchmarks.
1. Introduction
Image assessment models use visual features to score technical quality or aesthetic appeal, but preprocessing can obscure original-image geometry and composition. MetaRefine combines this context with existing models’ multi-level features and scores to refine predictions.
1. Introduction
- Existing assessment models share a visual-encoder and scoring-head design, with research improving multi-level fusion, spatial processing, and parameter-efficient adaptation.Image quality assessment concerns technical quality, while image aesthetic assessment concerns photographic appeal.Source: §1 · p. 1
- Resizing can remove original dimensions or alter proportions, while cropping and patch sampling can remove global composition context.Image size and resolution can affect aesthetic judgments and perceived technical quality, and aspect ratio is associated with aesthetic preference.Source: §1 · p. 2
- Prediction-error patterns vary across original-image aspect ratios in several models, motivating the combination of geometric context with their encoded visual evidence.The paper examines error variation by geometry before proposing score refinement.Source: §1 · p. 2
- MetaRefine combines aggregated multi-level visual features and the original score with log pixel area and log aspect ratio in lightweight residual heads.The framework is designed to apply across pretrained IQA and IAA architectures.Source: §1 · p. 2
- MetaRefine reports mean relative gains of 7.55% PLCC and 7.10% SRCC, plus reductions of 23.27% MAE and 21.93% RMSE.The framework also improves state-of-the-art IQA and IAA models.Source: §1 · p. 2
2. Related Work
Related work spans general-purpose no-reference IQA, specialized quality assessment, and image aesthetic assessment, using varied datasets and visual or learning strategies.
Image Quality Assessment
- No-reference IQA covers synthetic and authentic distortions, with methods using local quality weighting, ranking supervision, multi-scale or multi-level features, and vision-language quality descriptions.Examples include KADID-10k, KonIQ-10k, SPAQ, and LIVE-FB for general-purpose assessment.Source: §2 · p. 3
Other Image Quality Assessment
- Specialized IQA addresses panoramic, facial, and AI-generated images, with methods incorporating viewport sequences, facial landmarks, or image-prompt correspondence.The cited datasets include CVIQ, CGFIQA-40k, and AGIQA-3K.Source: §2 · p. 3
Image Aesthetic Assessment
- IAA benchmarks include AVA, AADB, TAD66K, and PARA, while methods exploit rating distributions, rankings, themes, attributes, observer characteristics, and spatial representations.Examples of spatial-representation approaches include pooling, multi-scale processing, and attention.Source: §2 · p. 3
3. Methodology
MetaRefine refines a pretrained assessment model’s scores using its original score, multi-level visual features, and original-image geometry while keeping the model frozen. Residual heads learn image-dependent corrections with a joint regression and ranking objective.
3. Methodology
- The pretrained assessment model remains frozen while residual heads use its original score, multi-level visual features, and original-image geometry to predict corrections.The corrections are averaged and added on the refinement score scale before mapping back to the evaluation scale; the framework can use another pretrained model without modifying its architecture.Source: §3 · p. 3, §3.1 · p. 3, Figure 3 · p. 4
3.2. Multi-Level Visual Feature Aggregation
- Features from multiple stages of the frozen model’s original forward computation are pooled and concatenated into a visual representation for refinement.Extraction locations may span multiple branches and include final-stage token summaries.Source: §3.2 · p. 3
3.3. Geometric Context Modeling
- Original image dimensions recorded before preprocessing provide log pixel area and log aspect ratio as geometric context.Log pixel area describes image size, while log aspect ratio describes shape and landscape or portrait orientation.Source: §3.3 · pp. 3–4
3.4. Unbounded Residual Refinement
- Each unbounded residual head predicts an image-dependent correction from the normalized joint input, and inference averages corrections from independently trained heads.The average correction is added to the original score; the two geometric features contribute only 2Kd input-layer weights across K heads.Source: §3.4 · p. 4
3.5. Joint Regression and Ranking Objective
- The objective trains each head against human scores and sampled relative preferences, combining robust regression, correlation, pairwise ranking, and residual regularization.The residual penalty discourages large adjustments without imposing a hard magnitude limit, and the same objective and head design are used across assessment models.Source: §3.5 · pp. 4–5
4. Experiments and Results
MetaRefine is evaluated against frozen assessment-model scores across 100 configurations, then tested through component ablations. It improves aggregate metrics with low median parameter overhead, while the ablations examine which inputs and pathways contribute to refinement.
4.1. Experimental Setup
- The evaluation compares refined and original scores on the same images across 51 IQA and IAA methods, 100 model-dataset configurations, and 21 datasets.Baselines retain their configured preprocessing and evaluation context; metrics are PLCC, SRCC, MAE, and RMSE.Source: §4.1 · p. 5
4.2. Comparison with Existing Methods
- Across 100 configurations, mean relative gains were 7.55% PLCC and 7.10% SRCC, while mean relative reductions were 23.27% MAE and 21.93% RMSE.Both correlations improved in 84 configurations, and all four metrics improved in 77; median parameter overhead was 1.11%.Source: §4.2 · pp. 5, 8, Table 1 · p. 6, Table 2 · p. 7
4.3. Ablation Study
- Removing visual features, geometric metadata, or the head’s original-score input reduces performance; removing the score skip connection lowers PLCC and SRCC by 15.75% and 14.28% on average.Removing the skip connection also increases MAE and RMSE in every ablation configuration; Table 3 summarizes relative changes against Full.Source: §4.3 · p. 8, Table 3 · p. 8
5. Conclusion
MetaRefine refines pretrained image-assessment scores by combining multi-level visual features and original scores with geometric metadata, while leaving the original scoring pipeline intact. Across 100 configurations on 21 datasets, it improves all four reported metrics on average with a median parameter overhead of 1.11%.
5. Conclusion
- Across 100 configurations on 21 datasets, PLCC and SRCC rise by 7.55% and 7.10% on average, while MAE and RMSE fall by 23.27% and 21.93%.The reported median parameter overhead is 1.11%, and gains include further improvements to SOTA IQA and IAA models.Source: §5 · p. 8
- MetaRefine combines multi-level visual features and original scores with geometric metadata, then learns residual corrections while preserving the original scoring pipeline.Only the residual heads are trained, allowing the same refinement design to accommodate diverse assessment architectures.Source: §5 · p. 8
- Ablations highlight contributions from multi-level visual features, geometric metadata, and original-score pathways; the paper concludes that MetaRefine reuses learned representations to improve score prediction.Source: §5 · p. 8
A. Further Method Details
The appendix details how MetaRefine aggregates features, normalizes joint inputs, samples training pairs, and interprets residual corrections. It also gives the scaling of added trainable parameters.
A.1. Feature Aggregation
- The fixed aggregation operator converts selected visual representations into vectors using spatial or token pooling, with image-level aggregation retaining the model’s original scoring computation.Aggregation can include frozen normalization or projection layers and combine original crops, scales, or viewports.Source: §A.1 · p. 12
A.2. Score Scales and Input Normalization
- The affine map T(s) = as + b sets the refinement score scale, while joint-input coordinates use training-set means and standard deviations, a positive lower bound, and clipping.Stored training statistics are reused at inference to balance visual activations, geometric features, and scores before the residual heads.Source: §A.2 · p. 12
A.3. Pair Sampling and Loss Functions
- Pairwise training samples distinct image pairs with a minimum label separation; the objective gives equal weight to each retained pair and combines soft preferences with binary cross-entropy and Huber regression.The Huber term is scaled by κ in the main paper’s regression term.Source: §A.3 · p. 12
A.4. Residual-Correction Interpretation
- The unconstrained correction minimizing expected squared error over functions of z is δ*(z) = E[r | z], where r is the frozen model’s residual on the refinement score scale.The identity describes population squared-error reduction from predictable residual variation; geometry can help when it explains residual variation beyond visual features and the original score.Source: §A.4 · p. 12
A.5. Trainable Parameter Count
- For K residual heads of hidden width d and input dimension D, the added parameter count is Nadd = K[d(D + 1) + (d + 1)].With a fixed head count, this count scales with D and d, without a separate dependence on frozen-model depth.Source: §A.5 · p. 12
B.1. Refinement Architecture and Optimization
The appendix specifies the residual-head training configuration, input and loss settings, pair sampling, and model-specific feature extraction. It also describes inference reuse, ablation feature groups, and parameter-overhead accounting.
B.1. Refinement Architecture and Optimization
- Three independently initialized residual heads use hidden width 64, output scale γ = 0.375, dropout 0.5, and 85 full-batch AdamW updates; their zero-initialized output layers predict unbounded corrections.Only residual-head parameters are updated, and the three corrections are averaged equally.Source: §B.1 · p. 12
B.2. Input Statistics and Score Scales
- Input statistics are computed from the residual-head training set, standard deviations are floored at 0.01, and standardized inputs are clipped to [−10, 10].Normalized-scale configurations use T(s) = 1 + 9s; others use the identity map, and refined scores are returned without dataset-range clipping.Source: §B.2 · p. 13
B.3. Loss Settings
- The loss settings use Huber transition κ = 0.2, smoothing constant ϵ = 0.02, and weights λabs = 0.05, λcorr = 0.5, λpair = 0.05, and λres = 0.005.Pairwise ranking uses temperature Trank = 0.5.Source: §B.3 · p. 13
B.4. Pair Sampling
- Pair sampling draws candidate image-index pairs uniformly with replacement, removes identical or insufficiently separated labels, and retains up to 16,384 valid pairs per sampling step.Pairs are resampled initially and after 20, 40, 60, and 80 updates; all heads share the schedule.Source: §B.4 · p. 13
B.5. Feature Extraction and Evaluation Context
- Feature extraction locations are architecture-specific, and aggregation preserves each model’s original scoring procedure across crops, scales, or viewports.The appendix gives examples including multi-branch features and MUSIQ’s final patch-token mean alongside class-token features.Source: §B.5 · p. 13
B.6. Feature Groups for Ablation
- Ablation feature groups follow architecture-specific extraction depths rather than equal network-depth divisions; removing a group zeros its normalized coordinates during training and inference.For CLIP-IQA+ (ViT-L/14), early, middle, and late groups use Transformer blocks 6, 12, and 18, while the final group contains the normalized scoring embedding.Source: §B.6 · p. 13
B.7. Computation and Inference
- At inference, all heads reuse features from one original-pipeline execution, apply training normalization statistics, disable dropout, and average the three corrections equally.Mean added parameters are 1.64M across the main-paper configurations, while the ablation subset’s default head cost averages 1.70M MACs because its mean input dimension differs.Source: §B.7 · p. 13
C.1. Parameter Overhead
- Parameter overhead divides added trainable parameters by the complete frozen model’s parameter count; UNIQUE-mix reaches 225.87% because high-dimensional features pair with a relatively small model.Source: §C.1 · p. 14
C.2. Data and Checkpoint Protocols
The protocols specify dataset partitions and frozen-model checkpoints, preserve model-specific evaluation settings, and define matched score scales and resampling procedures for evaluation and ablations.
C.2. Data and Checkpoint Protocols
- Flickr-AES uses a public 4,737-image test list; the remaining images are grouped by identical pixel content into training and validation splits of 32,186 and 3,576 images.The selected assessment models are frozen, and residual heads use their designated training splits.Source: §C.2 · p. 14
C.3. Model Reproduction and Adaptation
- Retrained configurations follow task-specific protocols before freezing: AMFF-Net is randomly initialized, while IQCaption360 is adapted to CVIQ and NIMA uses disjoint TID2013 reference groups.IP-IQA and AMFF-Net use AGIQA-3K; the passage also specifies training, validation, and evaluation counts for these adaptations.Source: §C.3 · pp. 14–15
C.4. Score Scales and Uncertainty
- Original and refined scores use identical label conversions and evaluation scales within each configuration; paired 95% intervals use 2,000 resamples of relevant groups.PLCC is computed directly from predicted scores without fitting a logistic mapping to evaluation labels.Source: §C.4 · p. 15
D.1. Experimental Setup
- Ablation variants use fixed partitions, matched training data and cached representations, and shared optimization settings; the aggregate reference is Full.The setup specifies seed groups and distinguishes the three-head aggregate comparison from matched-seed single-head studies.Source: §D.1 · p. 15
D.2. Input Features and Training Components
The ablations examine feature and training components, original-score pathways, head count and architecture, correction bounds, and uncertainty across seeds and evaluation samples.
D.2. Input Features and Training Components
- 0.313% and 0.325%: removing hidden dropout raises MAE and RMSE, while improving PLCC and SRCC by 0.533% and 0.085%, respectively.Full improves both correlations over every variant for TReS (FLIVE); it improves all four metrics in 12 comparisons each for TReS, PIAA-MIR, CLIP-IQA+, and MUSIQ, and 9 for TOPIQ.Source: §D.2 · p. 15
D.3. Original-Score Pathways
- 15.751% and 14.277%: removing the score skip lowers mean PLCC and SRCC; MAE and RMSE rise by 170.260% and 154.029%.Both no-skip variants increase both error metrics in every ablation configuration; removing only the head-score input produces mean relative changes from −0.037% to 0.004%.Source: Table S2 · p. 16, §D.3 · p. 16
D.5. Residual-Head Architecture
- 1.70M dense MACs per image: width 64 uses half the 3.40M required by width 128, while doubling width yields mean relative gains of 0.337% PLCC, 0.228% SRCC, 0.450% MAE, and 0.383% RMSE.The paper describes width 64 as retaining performance close to the wider head at lower computational and parameter cost.Source: §D.5 · p. 18
D.4. Number of Residual Heads
- Single-head results are compared with ensembles averaging one, three, five, or ten residual heads; only residual-head parameters are replicated.Single-head means and sample standard deviations use five seeds, while ensembles average predictions.Source: §D.4 · pp. 16–17, Table S5 · p. 19
D.6. Correction-Bound Sensitivity
- −1.38% PLCC, −1.35% SRCC, −45.27% MAE, and −40.41% RMSE: the bounded Huber control at τ = 0.375 has negative mean changes from Full.Negative values indicate lower correlations or higher errors than Full; Full outperforms this control on all four metrics in 56 configurations.Source: Table S8 · p. 20, §D.6 · p. 20
D.7. Stability and Uncertainty
- Ten-head results are compared with the default three-head Full using 2,000 paired bootstrap resamples; these intervals measure evaluation-sample uncertainty, distinct from initialization variability.The reported intervals are 95% percentile intervals without multiple-comparison adjustment.Source: §D.7 · p. 20
D.8. Performance Decreases and Metric Trade-offs
The reported decreases show that refinement can affect correlation and absolute error differently across model-dataset cases. The section also specifies end-to-end inference and timing protocols.
D.8. Performance Decreases and Metric Trade-offs
- WaDIQaM-NR on KonIQ-10k has MAE decrease from 0.8095 to 0.2740 while SRCC decreases from 0.8046 to 0.3557.SIND on KADID-10k likewise has MAE decrease from 1.5902 to 0.2740 while SRCC decreases from 0.8606 to 0.7634.Source: §D.8 · p. 20
- Charm on BAID has PLCC rise from 0.3833 to 0.3944 and SRCC from 0.3731 to 0.3778, while MAE rises from 0.0463 to 0.0723 and RMSE from 0.0633 to 0.0880.LAR-IQA KAN on UHD-IQA declines on all four metrics.Source: §D.8 · pp. 20–21
- Linear heads and a correction bound of 3.375 improve mean PLCC and SRCC relative to Full but increase MAE and RMSE.The comparisons indicate that head architecture and correction parameterization affect the balance between correlation and absolute score accuracy.Source: §D.8 · p. 21
E.1. Complete Inference Pipeline
- Inference efficiency is measured on an RTX 3090 at batch size one, retaining each original model’s crops, scales, or viewports and including feature extraction and transfers.Baseline measurements compute original scores without extracting the additional features used by residual heads.Source: §E.1 · p. 21
E.2. Timing and Memory Protocol
- The timing protocol uses eight fixed images, three interleaved measurement rounds, two warm-ups and eight timed calls per round; latency is reported as the median.Peak allocated and reserved GPU memory include the assessment model and residual heads, but exclude driver context and other processes.Source: §E.2 · p. 21
E.3. Training and Caching
The appendix separates training, cache, and end-to-end inference costs, with timings tied to specified runs and measurement protocols.
E.3. Training and Caching
- One single-head training run uses seed 314, the full training cache, and 85 updates; timing includes pair generation and optimization.Cache loading is timed separately from feature normalization and GPU transfer.Source: §E.3 · p. 21
- End-to-end inference comparisons cover the frozen model and one-, three-, five-, and ten-head refinement, using batch size one and 24 timed image calls per setting.The protocol reports median latency and images per second, with peak PyTorch allocated and reserved memory.Source: Table S10 · p. 22
- Training-feature storage counts only the visual feature values used by the full residual head, at four bytes per FP32 value; stored cache size counts retained feature-cache files.Cache-write timing includes writing to the operating system’s file cache without waiting for all data to reach disk.Source: §E.3 · p. 21, Table S11 · p. 22
F.1. Cross-Dataset Qualitative Comparisons
Figures S1–S6 show six illustrative image-level comparisons per dataset, pairing each assessment model’s original and MetaRefine-refined scores with ground truth. The examples span six datasets and use the dataset or evaluation score scales indicated for each grid.
F.1. Cross-Dataset Qualitative Comparisons
- Each grid pairs an assessment model’s original and refined scores for the same image with ground truth, complementing the paper’s quantitative results.Red and green mark original and refined scores, respectively, while blue marks ground truth.Source: §F.1 · p. 22
- AVA, KonIQ-10k, and SPAQ examples use the AVA 1–10 scale or normalized 0–1 scales, and SPAQ models retain their own original scoring pipelines.The examples show six illustrative comparisons per dataset.Source: §F.1 · p. 22, Figure S1 · p. 23, Figure S2 · p. 23
- KADID-10k examples span multiple contents and distortion types, while TAD66K scores are normalized to 0–1 and Charm’s scores are divided by 10; AADB examples use a 0–1 scale.Colors and paired-score notation follow Figure S1 for the datasets whose captions refer to it.Source: §F.1 · p. 22, Figure S4 · p. 23, Figure S5 · p. 24