PaperlayerView exact revision on arXiv

Source-linked AI summary

Next Token Is Enough: Realistic Image Quality and Aesthetic Scoring with Multimodal Large Language Model

Mingxing Li, Rui Wang, Lei Sun, Yancheng Bai, Xiangxiang Chu

arXiv:2503.06141v1cs.CV

TL;DR

UGC image scoring needs to reflect hierarchical human perception, while numerical-score prediction remains unresolved for MLLMs. The paper introduces RealQA with fine-grained attributes and studies next-token numerical scoring; attribute-guided CoT surpasses SOTA on five IQA/IAA datasets and generalizes to VQA.

  • Problem

    A single MOS misses hierarchical human perception, while how MLLMs should output numerical scores remains an open question.

  • Method

    The paper introduces RealQA, a 14,715-image dataset with 10 fine-grained attributes, and combines attribute extraction with next-token numerical scoring and CoT.

  • Results

    Attribute-guided CoT surpasses SOTA on five IQA and IAA datasets and shows strong zero-shot VQA generalization, including a +36.4% SRCC improvement on KoNViD.

  • Takeaways & Limitations

    Fine-grained attributes provide an interpretable route for MLLM-based image quality and aesthetic scoring while supporting transfer from image to video assessment.

  • Takeaways & Limitations

    RealQA retains ambiguous answers and focuses on common general attributes rather than separating artistic attributes.

Abstract

from arXiv · show

The rapid expansion of mobile internet has resulted in a substantial increase in user-generated content (UGC) images, thereby making the thorough assessment of UGC images both urgent and essential. Recently, multimodal large language models (MLLMs) have shown great potential in image quality assessment (IQA) and image aesthetic assessment (IAA). Despite this progress, effectively scoring the quality and aesthetics of UGC images still faces two main challenges: 1) A single score is inadequate to capture the hierarchical human perception. 2) How to use MLLMs to output numerical scores, such as mean opinion scores (MOS), remains an open question. To address these challenges, we introduce a novel dataset, named Realistic image Quality and Aesthetic (RealQA), including 14,715 UGC images, each of which is annoted with 10 fine-grained attributes. These attributes span three levels: low level (e.g., image clarity), middle level (e.g., subject integrity) and high level (e.g., composition). Besides, we conduct a series of in-depth and comprehensive investigations into how to effectively predict numerical scores using MLLMs. Surprisingly, by predicting just two extra significant digits, the next token paradigm can achieve SOTA performance. Furthermore, with the help of chain of thought (CoT) combined with the learnt fine-grained attributes, the proposed method can outperform SOTA methods on five public datasets for IQA and IAA with superior interpretability and show strong zero-shot generalization for video quality assessment (VQA). The code and dataset will be released.

1 Amap, Alibaba Group 2 BUPT

Figure 1 presents the RealQA dataset, its fine-grained attributes, their CoT use for public datasets, and real-world applications.

  • RealQA organizes 10 fine-grained attributes for image quality and aesthetic assessment, then applies them through CoT and composite scores.The figure also contrasts rankings from an AVA-trained Q-Align model with those from a RealQA-trained model.

1. Introduction

The paper addresses the need for perception-aligned UGC image scoring by combining hierarchical attributes with numerical-score prediction in MLLMs. It introduces RealQA and reports strong results from next-token scoring and attribute-guided CoT.

  • A single MOS cannot capture hierarchical perception, while textual labels and hidden-layer regression leave numerical scoring in MLLMs unresolved.The paper motivates attribute-level content for interpretability and more robust alignment with human perception.
  • Predicting two extra significant digits with the next-token paradigm can achieve superior numerical-score performance.The paper also studies numerical continuity to assess whether MLLMs understand scores as wholes rather than memorizing positional tokens.
  • Fine-grained attributes are extracted through cold-start training and combined with numerical scores in CoT training for public datasets.The resulting representations are intended to improve interpretability while supporting quality and aesthetic scoring.
  • +1.8% PLCC on Koniq-10k and +4.5% on LIVE Challenge over Q-Align, while image-only training improves KoNViD SRCC by +36.4%.The CoT method combines self-labeled fine-grained attributes with final scores and surpasses SOTA on five IQA and IAA datasets.
  • 14,715 UGC images in RealQA receive 10 fine-grained attributes spanning low-, middle-, and high-level human perception.The attributes support quality and aesthetic assessment beyond a single score.

2. Related Work

Related work connects IQA and IAA through shared perceptual attributes and reviews MLLM approaches to quality and aesthetic scoring. It highlights the limitations of independent task modeling, discrete textual levels, and regression-based scores.

  • 2.1. Connection between IQA and IAA: IQA and IAA are complementary because aesthetic properties can affect quality ratings and quality defects can reduce aesthetic appeal.Recent AIGC studies therefore explore jointly learned representations, while RealQA integrates both tasks through fine-grained attributes.
  • 2.2. Quality and Aesthetic Scoring with MLLMs: MLLMs have advanced IQA and IAA scoring, but numerical-score modeling remains open because discrete labels limit refinement and regression lacks natural next-token integration.Q-Align uses discrete levels, whereas Videoscore regresses scores from a linear layer.

3. Method

The method combines a hierarchical RealQA dataset, numerical-score modeling with continuous expectations, and a two-stage attribute-plus-CoT training recipe for MLLMs.

  • 3.1. Realistic Image Quality and Aesthetic Dataset: RealQA organizes image quality and aesthetics into high-, middle-, and low-level attributes, with filtering, annotation, and dataset splitting procedures.The dataset contains 14,715 UGC images, and annotations include high-level scores, multi-tier classifications, MLLM labeling, and annotator refinement.
  • 3.2. Next Token Prediction Paradigm: The next-token formulation exposes numerical ambiguity because cross-entropy supervises token likelihoods rather than the continuous distance between predicted and ground-truth scores.A numerically closer prediction can receive a larger loss, particularly when scores are represented by multiple tokens.
  • 3.2. Next Token Prediction Paradigm: NCM converts digit-token logits into continuous expectations and compares the expected predicted score with the ground-truth score using mean squared error.The method represents each digit with an expectation over the digit vector and combines digit expectations using numerical weights.
  • 3.2. Next Token Prediction Paradigm: NCM* excludes the ground-truth digit when calculating expectations, testing whether probability mass remains concentrated near the target digits.The variant uses a binary mask to suppress the ground-truth position and is intended to distinguish well-trained from poorly converged models.
  • 3.3. Training Recipe: The training recipe first cold-starts MLLMs to extract fine-grained attributes, then self-labels public datasets and retrains with attributes and numerical scores in a CoT format.The recipe uses multiple conversation formats for attribute extraction, direct scoring, and natural-language or regex-friendly CoT responses.
  • 3.3. Training Recipe: The model uses LoRA to fine-tune Qwen2-VL-7B while unfreezing the vision encoder and setting the LoRA rank to 128.The architecture is designed to learn image representations at different levels of granularity.

4. Experiments

The experiments evaluate RealQA-based fine-grained attribute extraction and MLLM numerical scoring across IQA, IAA, and VQA settings. Next-token prediction with extra digits, attribute-aware training, and CoT achieves strong benchmark performance and generalization.

  • 4.3. Comparison with General MLLMs: The proposed model achieves the highest average rank of 1 for fine-grained attribute extraction, outperforming GPT-4o, Qwen-VL-Max, and GPT-4V.
  • 4.4. Ablation of Fine-grained Attributes Training: Mixing training granularities yields the best fine-grained attribute performance, while predicting high-level attributes first performs similarly to predicting low-level attributes first.
  • 4.5. Directly Predict Numerical Scores by MLLMs: Qwen2-VL-7B achieves the best results in numerical sorting, while LLaVA-1.5-7B performs poorly and is almost guessing, and mPLUG-Owl2-7B also obtains poor results.
  • 4.5. Directly Predict Numerical Scores by MLLMs: Predicting one extra decimal place raises AVA SRCC by 0.094 over integer prediction, while further digits provide only slight improvement; the default is M=3.
  • 4.5. Directly Predict Numerical Scores by MLLMs: During naive training, cross-entropy converges while NCM remains unstable, whereas LoRA fine-tuning makes NCM, NCM∗, and Lce converge together.
  • 4.6. Comparison with Regression Method: The next-token paradigm outperforms regression at equal training epochs, and regression remains inferior even with additional training steps.
  • 4.7. Quantitative results: On IQA, the method achieves state-of-the-art performance, improves KonIQ SRCC/PLCC by 0.9%/1.0% with CoT, and raises LIVE Challenge PLCC by 4.5% over Q-Align.
  • 4.7. Quantitative results: On IAA, CoT improves SRCC/PLCC by 1.9%/1.8% on AVA, while the method exceeds UNIAA by 1.9% PLCC on cross-dataset TAD66K evaluation.

5. Conclusion

The paper introduces RealQA and concludes that fine-grained attributes and next-token numerical prediction support quality and aesthetic assessment. Its appendix details a numerical-sorting test used to evaluate MLLM handling of ordered decimals.

  • 5. Conclusion: RealQA contains 14,715 real-world images, each annotated with 10 fine-grained attributes, while two extra significant digits enable strong next-token scoring performance.
  • 5. Conclusion: CoT combined with fine-grained attributes surpasses state-of-the-art methods on five IQA and IAA datasets and generalizes strongly to VQA.
  • Appendix A. Details of Toy Example: The toy experiment sorts 10 randomly generated decimals over 200 repetitions using accuracy, recall, and hallucination metrics.
  • Appendix A. Details of Toy Example: The numerical-sorting metrics measure sequence-order correctness, the fraction of ground-truth numbers recovered, and the fraction of predicted numbers absent from the ground truth.

B.1. Discrepancy with IQA/IAA

The appendix evaluates RealQA’s transfer beyond standard IQA and IAA datasets and documents the annotation scheme used for interpretable assessment. Results indicate task-specific discrepancies and attribute-level detail.

  • B.1. Discrepancy with IQA/IAA: Training on KonIQ-10k or AVA and testing on RealQA gives the proposed method a 2% SRCC advantage over Q-Align.
  • B.1. Discrepancy with IQA/IAA: Training directly on RealQA significantly increases SRCC and PLCC, verifying a discrepancy between IQA/IAA tasks and UGC image assessment.
  • B.2. Annotation Granularity: RealQA attributes include 1–10 composition and eye-catching scores plus categorical labels for clutter, subject integrity, level shot, clarity, exposure, and saturation.
  • B.2. Annotation Granularity: The fine-grained attributes provide detailed image understanding and interpretable natural-language explanations for quality and aesthetic assessment.

B.4. Limitations

The appendix identifies ambiguity in some RealQA annotations and limits the dataset to common, general attributes rather than artistic attributes.

  • B.4. Limitations: RealQA retains ambiguous answers despite data cleaning and MLLM-output correction, including judgments of whether an image is horizontally shot.
  • B.4. Limitations: The dataset excludes artistic attributes because they require contextual understanding that is difficult to collect and annotate.

C.1. Numerical Scoring for MLLMs

The section examines how MLLMs handle numerical scoring and introduces tokenizer-aware analysis for Qwen2-VL. It reports numerical-sorting behavior, fine-grained attribute training formats, and adaptations needed across tokenizers.

  • C.1. Numerical Scoring for MLLMs: Qwen2-VL-7B generally sorts numbers correctly, but still produces occasional errors such as hallucinated red numbers.The experiment includes repeated numbers and values with different significant digits.
  • C.1. Numerical Scoring for MLLMs: Fine-grained attribute training uses separate conversations for attributes or levels, and a mixed conversation for all attributes.The templates distinguish attribute-level, hierarchical-level, and all-attribute prediction formats.
  • C.1. Numerical Scoring for MLLMs: Direct numerical scoring remains challenging because numerical values are composed of multiple tokens rather than a single classification target.The paper motivates numerical-scoring research by noting that token sequences such as 3.99 are generated across several tokens.
  • C.2. NCM for Qwen2-VL: NCM and NCM* monitor numerical understanding through Qwen2-VL’s tokenizer to test whether numbers are memorized positionally or understood as wholes.The methods are designed around tokenizer behavior during numerical prediction.
  • C.2. NCM for Qwen2-VL: Different tokenizers can split the same number differently, so NCM and NCM* require implementation changes while retaining their core ideas and theoretical basis.For example, 3.99 may be tokenized as 3, .9, 9 in another model, requiring corresponding adjustments to the mathematical-expectation calculations.

Appendix D. Visualization Results

The visualization results compare Q-Align trained on AVA with the proposed method trained on RealQA. On daily-user images, the proposed method produces better quality and aesthetic rankings.

  • Appendix D. Visualization Results: The proposed RealQA-trained method achieves better quality and aesthetic ranking results than AVA-trained Q-Align on daily-user images.The comparison covers real-world visualization results shown in Figures 3–8.
Loading 2503.06141v1…