Source-linked AI summary
Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels
Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, Qiong Yan, Xiongkuo Min, Guangtao Zhai, Weisi Lin
TL;DR
Visual scoring needs models that align with human judgments across diverse image and video content, while conventional MOS regression and direct-score alignment have limited robustness. Q-ALIGN emulates human rating with discrete text-defined levels and converts level probabilities into scores, achieving state-of-the-art results across IQA, IAA, and VQA and enabling ONEALIGN.
Problem
Existing visual scorers can perform well on specific datasets but struggle with out-of-distribution content, while LMMs still fail to predict human-consistent scores reliably.
Method
Q-ALIGN converts MOS labels into text-defined rating levels for LMM instruction tuning and converts predicted level probabilities into scores by weighted averaging.
Results
Q-ALIGN achieves state-of-the-art performance across IQA, IAA, and VQA under one general structure, and ONEALIGN unifies all three tasks in one model.
Takeaways & Limitations
Discrete-level alignment provides a unified approach for robust visual scoring and supports stronger generalization on unseen datasets.
Takeaways & Limitations
The reported LMM behavior evidence comes from five models not explicitly trained for visual rating tasks.
Abstract
from arXiv · showhide
The explosion of visual content available online underscores the requirement for an accurate machine assessor to robustly evaluate scores across diverse types of visual contents. While recent studies have demonstrated the exceptional potentials of large multi-modality models (LMMs) on a wide range of related fields, in this work, we explore how to teach them for visual rating aligned with human opinions. Observing that human raters only learn and judge discrete text-defined levels in subjective studies, we propose to emulate this subjective process and teach LMMs with text-defined rating levels instead of scores. The proposed Q-Align achieves state-of-the-art performance on image quality assessment (IQA), image aesthetic assessment (IAA), as well as video quality assessment (VQA) tasks under the original LMM structure. With the syllabus, we further unify the three tasks into one model, termed the OneAlign. In our experiments, we demonstrate the advantage of the discrete-level-based syllabus over direct-score-based variants for LMMs. Our code and the pre-trained weights are released at https://github.com/Q-Future/Q-Align.
1. Introduction
Visual scoring methods must generalize across diverse content and scoring scenarios, yet MOS-regression approaches often fail out of distribution. Q-ALIGN addresses this by teaching LMMs with discrete text-defined rating levels, achieving strong performance across visual scoring tasks and enabling unified assessment.
- Motivation: MOS-regression methods can perform well on specific datasets but often generalize poorly to novel content and mixed scoring scenarios.Their limited capacity cannot fully model the complicated factors affecting final visual scores.
- Approach: Q-ALIGN converts existing MOS labels into five text-defined levels and uses them as instruction-tuning targets for LMMs.The levels are excellent, good, fair, poor, and bad.
- Approach: During inference, Q-ALIGN extracts close-set probabilities for rating levels and computes the predicted score by weighted averaging.This emulates converting distributions of human ratings into MOS.
- Results: Q-ALIGN reaches state-of-the-art performance on 12 datasets spanning IQA, IAA, and VQA, with especially notable gains on unseen datasets.The method uses existing scores while training with less information than direct-score targets.
- Results: Q-ALIGN can remain competitive with 1/5 of IQA data or 1/10 of IAA data and supports combining datasets across tasks.These properties motivate ONEALIGN, a unified model for IQA, IAA, and VQA.
- Contributions: The discrete-level syllabus outperforms the score-based variant by +10% and supports a family of more capable visual assessors.The paper also proposes ONEALIGN as a unified model across the three visual scoring tasks.
2. Related Works
Prior visual scoring work spans IQA, IAA, VQA, and early LMM-based scoring strategies, but these approaches differ in task complexity and modeling structure. Q-ALIGN builds on LMM visual-language capabilities while emulating human rating and MOS post-processing.
- Image Quality Assessment (IQA): IQA evaluates how distortions and other quality issues affect human perception, using methods ranging from handcrafted features to end-to-end deep networks.Representative methods include NIMA, DBCNN, and HyperIQA.
- Image Aesthetic Assessment (IAA): IAA considers quality alongside higher-level attributes such as content, lighting, color, and composition.Q-ALIGN reportedly outperforms CLIP-based approaches without extra pre-training.
- Video Quality Assessment (VQA): VQA scores videos whose judgments can depend on quality, content, and aesthetics, making video input difficult to handle with image-scoring structures.Recent deep-learning methods improve VQA performance and OOD generalization, while CLIP-based approaches may rely on frame-by-frame inference.
- LMMs for Visual Scoring: Q-Bench and Q-Instruct explored LMM-based visual scoring through binary softmax pooling and low-level visual question-answering alignment.Q-ALIGN extends these insights into a systematic syllabus that emulates human rating and MOS post-processing.
- Human Rating Process: Human scoring is organized into training raters, collecting ratings, and converting ratings into MOS.This three-stage structure motivates Q-ALIGN’s rating-level training and probability-based inference.
3. The Q-ALIGN
Q-ALIGN designs LMM visual scoring around the human opinion-collection process: train with text-defined levels, collect level-oriented predictions, and convert their distribution into a score. LMMs’ preference for qualitative responses motivates this target design.
- Method Overview: Q-ALIGN’s methodology is organized around rating-level teaching, rating-level-to-score conversion, unified image-video structure, and task-specific conversation formats.These components are introduced as the framework for teaching LMMs to score.
- Human-Rating Analogy: Human-rater training aligns annotators with examples for each rating level without displaying precise scores.This process supplies the model-level analogy for discrete rating supervision.
- Human-Rating Analogy: Human annotators either choose rating levels or use level-guided sliders, without directly entering numerical scores.Their initial ratings are later averaged into MOS.
- Motivation: Qualitative rating levels can reduce cognitive load and improve subjective-study outcomes compared with directly eliciting numerical ratings.The paper connects this human tendency to its choice of supervision targets.
- LMM Rating Behavior: LMMs predominantly produce qualitative adjectives before task-specific alignment, so Q-ALIGN uses rating levels instead of scores as learning targets.This avoids requiring the model to learn numerical output formatting before learning visual scoring.
- Scope: The paper notes that its LMM-behavior observation concerns five models that were not explicitly trained for visual rating tasks.This bounds the stated pre-alignment behavior evidence.
3.2. Conversion between Rating Levels and Scores
Q-ALIGN converts existing scores into five discrete text-defined rating levels for training, then converts level probabilities back into scores during inference. The conversion preserves approximately 0.95 linear correlation with original scores on five training datasets.
- Training conversion: Q-ALIGN converts existing scores into five discrete rating levels during training.The levels are bad, poor, fair, good, and excellent.
- Training conversion: Scores are uniformly partitioned between the minimum m and maximum M into five equidistant intervals.Each interval is assigned one standard text-defined rating level.
- Conversion precision: Around 0.95 PLCC is retained by the score-to-level conversion across five training datasets.Table 2 reports SRCC/PLCC metrics for the conversion precision.
- Inference conversion: During inference, LMM logits for the five levels are normalized with a close-set softmax to obtain level probabilities.The probabilities are then used to recover a predicted score through weighted averaging.
- Inference conversion: The predicted score is obtained as a weighted average over probabilities assigned to reverse-mapped rating levels.This inference procedure is theoretically equivalent to post-processing human ratings and generalizes the binary good/poor softmax strategy.
3.3. Model Structure
Q-ALIGN uses an LMM structure with a visual abstractor that sharply reduces image tokens. This supports feeding image sequences for videos under the same general structure.
- Model structure: The visual abstractor reduces tokens per image from 1024 to 64.Under LLaMA2's 2048-token context, the structure can feed up to 30 images during supervised fine-tuning.
- Model structure: Q-ALIGN is built on the open-source mPLUG-Owl-2 LMM.The adopted model provides visual perception and language understanding components.
- Unified image-video structure: Images and videos are handled under one structure because videos are represented as sequences of images.The token reduction makes this unified treatment practical.
3.4. Conversation Formats
The conversation formats define how visual inputs and discrete rating levels appear in task-specific LMM instruction-response pairs.
- Conversation formats: The format denotes the image token as <img> and the converted rating level as <level>.
- Conversation formats: Task-specific exemplar conversations are defined using these image and level placeholders.
- Conversation formats: The formats support asking the model to evaluate visual content and respond with a rating level.
Image Quality Assessment (IQA)
The IQA conversation format presents an image to the assistant and requests a quality judgment expressed as a rating level.
- Image quality assessment: The user supplies an image using the <img> token.
- Image quality assessment: The prompt asks, “Can you evaluate the quality of the image?”
- Image quality assessment: The assistant responds that the image quality is <level>.
Image Aesthetic Assessment (IAA)
The IAA interaction asks an LMM to judge an image’s aesthetics using a text-defined rating level.
- The image-aesthetic prompt requests the assistant to complete a text-defined <level> response.
Video Quality Assessment (VQA)
The VQA interaction asks an LMM to rate video quality through a text-defined level, with paraphrased queries used for augmentation.
- The video-quality prompt requests the assistant to complete a text-defined <level> response.
- Randomly selected paraphrases augment user queries, while supervision is applied only to the LMM responses.
4. Experiments
Experiments evaluate Q-ALIGN across IQA, IAA, and VQA, including in-distribution, cross-dataset, mixed-data, few-shot, efficiency, and ablation settings. Results show strong accuracy, OOD generalization, data efficiency, unified multi-task performance, and an advantage over direct score training.
- IQA: Q-ALIGN improves IQA cross-dataset performance by more than 10% over visual-only methods, and by 8% and 4% over CLIP-IQA+ and LIQE.
- IQA: Using only 20% of IQA training images, FEWSHOT-Q-ALIGN reaches performance comparable to existing state-of-the-art approaches.
- IAA: Q-ALIGN outperforms LIQE by 7%, Aesthetic Predictor by 10%, and VILA by 6% on IAA, while FEWSHOT-Q-ALIGN surpasses existing methods using 10% of AVA.
- VQA: With sparse 1fps frames, Q-ALIGN outperforms specialized VQA approaches and surpasses FAST-VQA by 6% on LSVQ-to-MaxWell cross-dataset evaluation.
- ONEALIGN: ONEALIGN improves multi-task performance over single-task variants and raises OOD SRCC by 6.8% on AGIQA, 3.6% on CSIQ, 1.7% on LIVE, and 1.1% on KoNViD.
- Ablation studies: Level-based training improves cross-dataset performance by 10% on average over score-based training, including a 40% gain from in-the-wild to synthetic IQA.
5. Conclusion
Q-ALIGN teaches LMMs visual scoring with discrete text-defined levels rather than direct scores, improving IQA, IAA, and VQA under one structure. ONEALIGN further unifies these tasks into a single model.
- Q-ALIGN instructs LMMs with discrete text-defined levels rather than direct numerical scores.
- The approach improves IQA, IAA, and VQA under one general structure and unifies all three tasks in ONEALIGN.