PaperlayerView exact revision on arXiv

Source-linked AI summary

Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision

Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, Weisi Lin

arXiv:2309.14181v3cs.CVcs.AIcs.MM

TL;DR

MLLMs lack adequate evaluation of low-level visual perception and understanding despite their shift toward general-purpose intelligence. Q-Bench addresses this gap with benchmarks for perception, description, and quality assessment, finding preliminary but unstable and relatively imprecise low-level visual skills.

  • Problem

    Existing evaluation inadequately measures MLLMs’ low-level visual perception and understanding, motivating systematic assessment of these abilities.

  • Method

    Q-Bench combines LLVisionQA for low-level question answering, LLDescribe with GPT-assisted description comparison, and a unified softmax-based strategy for quantitative IQA prediction.

  • Results

    MLLMs show preliminary low-level visual skills across perception, description, and assessment, but these skills remain unstable and relatively imprecise.

  • Takeaways & Limitations

    Q-Bench provides a benchmark for discovering and enhancing the low-level visual abilities of general-purpose MLLMs.

  • Takeaways & Limitations

    The GPT-assisted description evaluation may still contain hallucinations, and judging agreement with gold descriptions remains subjective.

Abstract

from arXiv · show

The rapid evolution of Multi-modality Large Language Models (MLLMs) has catalyzed a shift in computer vision from specialized models to general-purpose foundation models. Nevertheless, there is still an inadequacy in assessing the abilities of MLLMs on low-level visual perception and understanding. To address this gap, we present Q-Bench, a holistic benchmark crafted to systematically evaluate potential abilities of MLLMs on three realms: low-level visual perception, low-level visual description, and overall visual quality assessment. a) To evaluate the low-level perception ability, we construct the LLVisionQA dataset, consisting of 2,990 diverse-sourced images, each equipped with a human-asked question focusing on its low-level attributes. We then measure the correctness of MLLMs on answering these questions. b) To examine the description ability of MLLMs on low-level information, we propose the LLDescribe dataset consisting of long expert-labelled golden low-level text descriptions on 499 images, and a GPT-involved comparison pipeline between outputs of MLLMs and the golden descriptions. c) Besides these two tasks, we further measure their visual quality assessment ability to align with human opinion scores. Specifically, we design a softmax-based strategy that enables MLLMs to predict quantifiable quality scores, and evaluate them on various existing image quality assessment (IQA) datasets. Our evaluation across the three abilities confirms that MLLMs possess preliminary low-level visual skills. However, these skills are still unstable and relatively imprecise, indicating the need for specific enhancements on MLLMs towards these abilities. We hope that our benchmark can encourage the research community to delve deeper to discover and enhance these untapped potentials of MLLMs. Project Page: https://q-future.github.io/Q-Bench.

1 INTRODUCTION

Q-Bench is introduced as a systematic benchmark for MLLMs’ low-level visual perception, description, and quantitative quality assessment. It defines three abilities and supports them with diverse datasets, expert descriptions, GPT-assisted evaluation, and a softmax-based scoring strategy.

  • The benchmark targets low-level abilities because they are associated with applications including recommendation, camera guidance, and visual quality enhancement.
  • LLVisionQA contains 2,990 images from 10 diverse sources, each paired with a question, correct answer, and false candidates.Its questions include Yes-or-No, What, and How formats, spanning distortion versus other attributes and global versus local perception.
  • LLVisionQA forms a balanced benchmark by combining three question types with four quadrants of low-level visual concerns.
  • LLDescribe uses expert-written long golden descriptions for 499 images to evaluate MLLM outputs for completeness, preciseness, and relevance with GPT assistance.The descriptions average 58 words and serve as reference texts for comparison.
  • For quality assessment, Q-Bench asks MLLMs to predict quantifiable scores aligned with human-rated mean opinion scores on low-level visual appearances.A softmax pooling strategy over logits for frequent response tokens addresses the difficulty of obtaining sufficiently quantifiable outputs.
  • Q-Bench evaluates three abilities: low-level attribute perception, natural-language description of low-level information, and quantitative image-quality assessment.

2 CONSTRUCTING THE Q-BENCH

Q-Bench is a focused benchmark for low-level visual abilities, covering perception, description, and assessment across diverse image appearances. It combines purpose-built datasets, GPT-assisted language evaluation, and a softmax-based strategy for obtaining quantifiable quality scores.

  • General principles: Q-Bench focuses on low-level image attributes without requiring reasoning or outside knowledge.
  • General principles: Two-thirds of perception and description images are in-the-wild, while the remainder come from artificial distortions or generative processes.
  • Low-level perception: 2,990 images form LLVisionQA, with one question, correct answer, and 1–3 false candidates per image.The dataset covers multiple sources and low-level concerns.
  • Low-level perception: LLVisionQA organizes low-level concerns along distortions versus other attributes and global versus local in-context perception.These axes distinguish technical image degradation from aesthetic-related attributes and overall from localized appearances.
  • Question types: Yes-or-No, What, and How questions probe judgments, attribute or context classification, and finer-grained non-extreme appearances.How questions can represent intermediate conditions such as medium clarity.
  • Low-level description: LLDescribe contains 499 images with expert-written long descriptions, and single-modal GPT evaluates MLLM outputs for completeness, preciseness, and relevance.The descriptions average 40 words in the benchmark overview and are designed to cover as many occurring low-level concerns as possible.
  • Quality assessment: The quality-assessment task uses existing IQA datasets, while a softmax-based strategy converts logits for good and poor into quantifiable scores.The resulting qpred outputs show higher correlation with human ratings in the reported experimental analysis.

3 RESULTS ON Q-BENCH

Q-Bench evaluates MLLMs across low-level perception, description, and assessment, finding promising but uneven performance. Stronger models lead perception and assessment, while low-level description remains limited and finer-grained assessment is difficult.

  • 15 MLLMs were evaluated on 13 popular open-source models and GPT-4V under zero-shot settings.
  • Perception: More than 60% accuracy on both LLVisionQA subsets was achieved by InternLM-XComposer-VL, LLaVA-v1.5, QWen-VL, and InstructBLIP (Flan-T5).
  • Perception: +9% performance gave GPT-4V a large margin over open-source MLLMs, while Senior-level Human accuracy remained 8% higher.
  • Description: InternLM-XComposer-VL performed best for description relevance, but nearly all MLLMs reached only an acceptable 0.8/2.0 standard for completeness and precision.
  • Assessment: 20% better performance than CLIP-ViT-Large-14 was achieved by the strongest MLLM on assessment, while finer-grained LIVE-FB and CGIQA-6K cases remained less accurate.
  • Assessment: Assessment used seven IQA datasets with at least 1,000 images and 15 human ratings per image.

4 CONCLUSION

Q-Bench benchmarks three distinct low-level visual abilities: accurate perception, precise and complete description, and quantitative quality assessment. Its evaluation finds decent abilities without low-level-specific training, but not yet reliable general low-level visual assistance.

  • 4 CONCLUSION: Q-Bench examines accurate perception, precise and complete description, and quantitative assessment of image quality.
  • 4 CONCLUSION: The benchmark combines two multimodal low-level-vision datasets with a unified softmax-based quantitative IQA strategy.
  • 4 CONCLUSION: Several extraordinary MLLMs show decent low-level abilities without low-level-specific training.
  • 4 CONCLUSION: Eleven photography experts annotated Q-Bench images in a controlled laboratory setting, with each expert labeling up to 30 images daily.
  • 4 CONCLUSION: LLVisionQA examples span distortions and other low-level attributes, including color, lighting, composition, and visual styles.

A.2.1 EVALUATION DETAILS FOR PERCEPTION ABILITY

The perception evaluation exposes a prompt-format failure for Kosmos-2 in multi-choice answering and tests close-set inference as a workaround. GPT evaluation uses repeated voting to reduce response variability.

  • Prompt failure: In 86% of primary-sample situations, Kosmos-2 appended a new candidate instead of selecting an offered answer.
  • Prompt failure: A prompt modification reduced Kosmos-2's prompt-failure rate to less than 10% in the sample set.
  • Close-set inference: Close-set inference ranks candidate answers by perplexity and selects the answer with minimum generative loss.
  • Close-set inference: Perplexity-based close-set inference significantly improved Kosmos-2 results after prompt failures were eliminated.
  • GPT evaluation: Five-round GPT voting improved evaluation accuracy from 93.2% to 98.4% in human analysis of a sample set.
  • GPT evaluation: GPT evaluation examples include rephrased, distracted, and both-answer responses, with voting yielding correct outcomes for the first two examples but not the third.

A.2.2 EVALUATION DETAILS FOR DESCRIPTION ABILITY

The evaluation uses GPT prompts to compare MLLM descriptions with golden low-level descriptions, scoring completeness, precision, and relevance. Five repeated GPT judgments are averaged to reduce response variability.

  • GPT Evaluation: The pipeline repeats each GPT prompt five times and averages the answers to mitigate variability in identical-prompt responses.The final score is the mean of the five outputs.
  • GPT Evaluation: GPT evaluates whether an MLLM description completely includes the low-level information in a golden reference description.The prompt assigns scores of 2 for complete or near-complete inclusion, 1 for partial or similar information, and 0 for no inclusion.
  • GPT Evaluation: The precision metric penalizes descriptions that contradict the reference on attributes such as blur, quality, colorfulness, noise, or exposure.Examples include describing a clear image as blurry or a dark image as bright.
  • Examples: The evaluation examples include a completeness score of 1.0 and a precision score of 0.0 from five identical GPT judgments.The completeness example receives five scores of 1, while the contradictory description receives five scores of 0.
  • GPT Evaluation: The relevance metric scores whether a description addresses low-level visual information, using 2 for complete relevance, 1 for partial relevance, and 0 for irrelevance.A dog-image example receives a five-round mean relevance score of 1.4.

A.2.3 EVALUATION DETAILS FOR ASSESSMENT ABILITY

The assessment evaluation converts MLLM token probabilities into quantitative image-quality scores and compares them with established visual-quality baselines and datasets. The protocol adapts token choices and dataset handling to each model or benchmark setting.

  • IQA Strategy: The softmax-based strategy enables causal MLLMs to quantitatively predict image quality from image-and-text inputs.The procedure is described as a short, easily integrated algorithm for new MLLMs.
  • Evaluation Protocol: The benchmark reports IQA performance using SRCC/PLCC across multiple MLLMs and IQA datasets, including comparisons with CLIP-ViT-Large-14.CLIP quality scores are derived from cosine similarities to prompts for good- and poor-quality images.
  • Model-Specific Settings: For Flan-T5-based InstructBLIP, the evaluation replaces the common good↔poor token pair with high↔low because those are its top-frequency quality tokens.The high↔low pair generally performs better than good↔poor across most databases.
  • IQA Strategy: The strategy computes quality from model probabilities for positive and negative quality tokens rather than relying only on argmax output.The synonym ensemble extends positive and negative token sets, such as good, fine, high versus poor, bad, low.
  • Evaluation Protocol: For CGIQA-6K, results are computed separately on 3,000 game images and 3,000 movie images before averaging, with baselines evaluated under the same protocol.NIQE and CLIP-ViT-Large-14 use the same split-specific procedure for fairness.

A.3.2 EXTENDED RESULTS FOR PERCEPTION

Extended perception results show that MLLMs remain unstable on basic low-level attributes, while contextual prompts can improve in-context answering and models show relative strength on what questions. Performance is similar across the dev and test subsets, though Yes-questions are generally easier than No-questions.

  • Dev-Subset Results: MLLMs perform similarly on the LLVisionQA dev and test subsets, supporting the dev results as a cross-validation reference.The dev subset is intended for future public release and had not been seen by the evaluated models.
  • Radar-Chart Analysis: Adding content context does not degrade perception performance and can improve answers to in-context questions.The authors interpret this as evidence that appropriate higher-level prompt context may improve low-level perception precision.
  • Radar-Chart Analysis: MLLMs show strong capability on what questions, suggesting potential reasoning ability in low-level perception.This observation is presented as a characteristic for future investigation and guidance-based improvement.
  • Yes-or-No Bias: All evaluated MLLMs achieve higher accuracy on Yes-questions than No-questions, with some models showing severe bias toward Yes answers.The LLVisionQA dataset contains 62% Yes-questions and 38% No-questions.
  • Qualitative Results: Qualitative examples show instability on basic attributes such as blur and failures on in-context questions that are easy for humans.These examples motivate further improvement of basic low-level perception.

A.3.3 EXTENDED RESULTS FOR DESCRIPTION Bar Chart for Different MLLMs:

Extended description results show that current MLLMs struggle to produce complete and accurate low-level descriptions, despite generally achieving higher relevance scores. Qualitative comparisons reveal disagreements about quality and low-level attributes, particularly for AI-generated images.

  • Bar-Chart Results: Current MLLMs struggle to describe complete and accurate low-level visual information.The description evaluation visualizes performance across three dimensions.
  • Bar-Chart Results: Relevance scores are generally higher than the other description dimensions, indicating that most MLLMs can follow the abstract description instruction.The reported bottleneck is perception of low-level attributes rather than instruction following.
  • Qualitative Comparisons: Different MLLMs diverge in their descriptions of image quality and related low-level attributes, especially on AI-generated images.Most models can describe image content precisely, even though content description was not instructed in the user prompt.

A.3.4 EXTENDED RESULTS FOR ASSESSMENT

Extended assessment results show that MLLMs can correlate with human quality ratings, especially for coarse quality differences, but remain less accurate for fine-grained high-quality images. The softmax strategy and synonym ensembling further improve quantitative IQA evaluation, although prompt preferences vary across models.

  • MLLMs notably correlate with human ratings and align very highly on relatively coarse situations such as AGIQA-3K and SPAQ.
  • MLLMs remain inaccurate on finer-grained datasets such as LIVE-FB and CGIQA-6K, where images are predominantly or entirely high quality.
  • Softmax vs Argmax: The softmax strategy outperforms argmax across all MLLMs and IQA datasets because argmax degenerates into only two scores, 0 and 1.
  • Token choices: The high↔low token pair is more competitive than good↔poor for InstructBLIP (Flan-T5-XL), indicating that token choice affects IQA performance.
  • Synonym Ensemble: Up to 2% accuracy improvement, averaging 1.3%, comes from synonym ensembling on the top-five MLLMs.
  • Prompt sensitivity: Different MLLMs favor different prompt combinations, although good↔poor is the overall best single word pair and remains the evaluation strategy.

B STATEMENT ON DATA CONTAMINATION

The benchmark treats contamination differently across its tasks: private perception and description materials limit exposure, while text-only quality knowledge is not considered image-data contamination for IQA.

  • The LLVisionQA test subset and all LLDescribe labels are private, while their questions, answers, and descriptions were unseen by existing MLLMs.
  • For tested open-source models, technical reports indicate that IQA datasets did not participate in multimodal training.
  • Textual knowledge linking blur with low quality is not considered IQA data contamination because language models cannot see the associated images.

C LIMITATIONS AND DISCUSSIONS

The discussion identifies biases and subjectivity in evaluation, and notes that Q-Bench measures low-level visual capabilities without directly guiding their improvement.

  • 62% of LLVisionQA questions are Yes-questions, and MLLMs frequently answer “yes,” creating potential comparison bias.
  • Description matching lacks an absolute standard, and the five-round GPT-assisted evaluation may still contain hallucinations.
  • Q-Bench provides comprehensive evaluation but no direct guidance for enhancing low-level visual capabilities.
Loading 2309.14181v3…