PaperlayerView exact revision on arXiv

Source-linked AI summary

Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision

Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, Weisi Lin

arXiv:2309.14181v3cs.CVcs.AIcs.MM

TL;DR

Evidence on MLLMs’ low-level visual abilities remains inadequate despite their relevance to applications. Q-Bench benchmarks perception, description, and quality assessment, finding that several models show decent low-level abilities without low-level-specific training.

  • Problem

    Evidence is inadequate on MLLMs’ low-level visual perception and understanding, abilities associated with applications including recommendation and camera guidance (Wu et al., 2023c; Zhang et al., 2022).

    Source: §1 · p. 2

  • Method

    Q-Bench evaluates perception with LLVisionQA, description with LLDescribe and GPT-assisted comparison, and image quality with a unified softmax-based prediction strategy.

    Source: §1 · pp. 2–3, §2.3 · p. 6

  • Results

    Several MLLMs show decent low-level visual abilities even without low-level-specific training.

    Source: §4 · p. 9

  • Takeaways & Limitations

    The authors argue that MLLMs should achieve accurate low-level perception, precise and complete description, and quantitative image-quality assessment.

    Source: §4 · p. 9

    Description scoring is subjective and may contain GPT hallucinations despite the five-round GPT-assisted evaluation protocol.

    Source: C LIMITATIONS AND DISCUSSIONS · p. 27

Abstract

from arXiv · show

The rapid evolution of Multi-modality Large Language Models (MLLMs) has catalyzed a shift in computer vision from specialized models to general-purpose foundation models. Nevertheless, there is still an inadequacy in assessing the abilities of MLLMs on low-level visual perception and understanding. To address this gap, we present Q-Bench, a holistic benchmark crafted to systematically evaluate potential abilities of MLLMs on three realms: low-level visual perception, low-level visual description, and overall visual quality assessment. a) To evaluate the low-level perception ability, we construct the LLVisionQA dataset, consisting of 2,990 diverse-sourced images, each equipped with a human-asked question focusing on its low-level attributes. We then measure the correctness of MLLMs on answering these questions. b) To examine the description ability of MLLMs on low-level information, we propose the LLDescribe dataset consisting of long expert-labelled golden low-level text descriptions on 499 images, and a GPT-involved comparison pipeline between outputs of MLLMs and the golden descriptions. c) Besides these two tasks, we further measure their visual quality assessment ability to align with human opinion scores. Specifically, we design a softmax-based strategy that enables MLLMs to predict quantifiable quality scores, and evaluate them on various existing image quality assessment (IQA) datasets. Our evaluation across the three abilities confirms that MLLMs possess preliminary low-level visual skills. However, these skills are still unstable and relatively imprecise, indicating the need for specific enhancements on MLLMs towards these abilities. We hope that our benchmark can encourage the research community to delve deeper to discover and enhance these untapped potentials of MLLMs. Project Page: https://q-future.github.io/Q-Bench.

1 INTRODUCTION

Although MLLMs have been explored mainly for high-level visual understanding, their low-level perception and understanding remain unclear despite their relevance to applications such as recommendation, camera guidance, and visual enhancement. Q-Bench addresses this gap with a systematic benchmark spanning low-level perception, description, and quality assessment.

Low-level perception

  • LLVisionQA benchmarks low-level attribute perception with 2,990 images from 10 sources, using three question types across four quadrants of concerns.The quadrants distinguish distortions from other attributes and global perception from local, in-context perception.Source: §1 · pp. 2–3

Low-level description

  • LLDescribe evaluates low-level descriptions against expert-written references for 499 images, scoring completeness, preciseness, and relevance through GPT-assisted comparison.The golden descriptions average 58 words each.Source: §1 · p. 3

Quality assessment

  • For IQA, Q-Bench predicts quality from softmax-pooled logits for “good” and “poor,” a strategy reported to correlate better with human perception than direct argmax token outputs.The resulting predictions are compared with human opinion scores on existing IQA databases.Source: §1 · p. 3

2 CONSTRUCTING THE Q-BENCH

Q-Bench evaluates MLLMs on low-level perception, description, and quality assessment while excluding tasks that require reasoning or outside knowledge. Its datasets and evaluation methods are designed to cover diverse low-level appearances and concerns.

2.2 BENCHMARK ON LOW-LEVEL PERCEPTION ABILITY

  • LLVisionQA contains 2,990 image-question-answer tuples and evaluates low-level perception using one question, one correct answer, and 1–3 false answers per image.The images come from multiple sources; the benchmark also uses k-means clustering to preserve diversity in sampled low-level attribute indicators, while assessment evaluates full images from seven IQA datasets.Source: §2.2 · p. 4, §2.1 · p. 4

2.2.1 QUADRANTS FOR LOW-LEVEL VISUAL CONCERNS

  • LLVisionQA organizes concerns along distortion versus other low-level attributes and global versus local in-context perception axes.Distortions directly degrade image quality, while other attributes need not; local questions require understanding image content or context, unlike global-perception questions.Source: §2.2.1 · p. 4

2.2.2 QUESTION TYPES

  • Its Yes-or-No, What, and How questions test judgments, identification of multiple attributes, and finer-grained descriptions of non-extreme appearances.Around 40% of Yes-or-No questions have “no” as the correct answer to reduce yes-response bias.Source: §2.2.2 · p. 5

2.2.3 GPT-ASSISTED EVALUATION PROCESS

  • Because MLLMs express correct answers in varied formats, LLVisionQA uses a five-round GPT-assisted process to judge correctness from questions, correct answers, and model replies.Correct and false answer options are shuffled during evaluation.Source: §2.2.3 · pp. 5–6

2.3.1 DEFINING Golden LOW-LEVEL DESCRIPTIONS FOR IMAGES

  • LLDescribe contains expert-provided golden low-level descriptions for 499 images spanning all 10 sources, designed to cover as many low-level concerns as possible when those concerns occur in the images.Source: §2.3 · p. 6, §2.3.1 · p. 6

2.3.2 EVALUATION WITH SINGLE-MODAL GPT

  • Single-modal GPT compares MLLM descriptions with LLDescribe references on completeness, preciseness, and relevance, scoring each dimension from 0 to 2 over five rounds.Completeness rewards matched information, preciseness penalizes contradictions, and relevance measures how much output concerns low-level information; the final score is a weighted average.Source: §2.3.2 · p. 6

2.4.1 WEAK MEASURABILITY OF MLLM OUTPUTS

  • On toy experiments, direct ratings were good (78%) and poor (20%), while numerical ratings concentrated on 5 (84%), 1 (9%), and 3 (5%).Scores 2 and 4 each appeared at less than 1%; outputs were biased toward positive and extreme judgments, with only two effective scales.Source: §2.4.1 · p. 7

2.4.2 A SOFTMAX-BASED EVALUATION STRATEGY

  • Softmax pooling over the good and poor token logits converts MLLM quality judgments into quantifiable scores that correlate more highly with human ratings.The strategy uses a simple direct instruction and applies across MLLMs; the paper reports higher correlation in its experimental analysis.Source: §2.4.2 · p. 7, §2.4 · p. 7

3 RESULTS ON Q-BENCH

Q-Bench evaluates 15 variants across 13 open-source MLLMs and GPT-4V in zero-shot settings, finding preliminary but uneven low-level perception, description, and quality-assessment abilities.

3 RESULTS ON Q-BENCH

  • The study evaluates 15 variants on 13 open-source MLLMs and GPT-4V under zero-shot settings.The perception evaluation uses LLVisionQA dev and test subsets.Source: §3 · p. 8, §3.1 · p. 8

3.1 RESULTS AND OBSERVATIONS ON PERCEPTION

  • InternLM-XComposer-VL led open-source LLVision accuracy; LLaVA-v1.5, QWen-VL, and InstructBLIP followed closely, with these four exceeding 60% on both subsets.Almost all MLLMs performed worse on distortions than on other low-level attributes; LLaMA-Adapter-V2 was the exception, and all MLLMs showed a yes-over-no response preference.Source: §3.1 · p. 8
  • GPT-4V outperformed open-source MLLMs by 9% and matched Junior-level Human accuracy, but Senior-level Human accuracy was 8% higher.GPT-4V, like open-source models, struggled to recognize distortions.Source: §3.1 · p. 8

3.2 RESULTS AND OBSERVATIONS ON DESCRIPTION

  • InternLM-XComposer-VL again ranked best for low-level description, especially relevance, but even the strongest MLLMs lacked excellent completeness and precision.Almost all MLLMs achieved an acceptable description score of 0.8/2.0, indicating limited and primary description ability.Source: §3.2 · p. 8

3.3 RESULTS AND OBSERVATIONS ON ASSESSMENT

  • InternLM-XComposer-VL, the top assessment model, outperformed CLIP-ViT-Large-14 by 20%, while most MLLMs surpassed NIQE on non-natural images.The evaluation covered seven IQA datasets with at least 1,000 images and 15 human ratings per image; synonym ensembling generally improved MLLM IQA accuracy, but finer-grained LIVE-FB and CGIQA-6K remained challenging.Source: §3.3 · p. 9

4 CONCLUSION … A.2 DETAILS ON BENCHMARK EVALUATION SETTINGS

Q-Bench evaluates low-level visual perception, description, and image-quality assessment, finding that some MLLMs show decent abilities without low-level-specific training but remain far from reliable general assistants. Its appendix details expert review and LLVisionQA’s attribute and question design.

4 CONCLUSION

  • Q-Bench assesses low-level perception, description, and quantitative image-quality assessment using two multimodal datasets and a unified softmax-based IQA strategy; several MLLMs perform decently without low-level-specific training but remain short of reliability.The authors hope the benchmark’s observations will spur stronger low-level perception and understanding in future MLLMs.Source: §4 · p. 9

A.1.1 SUBJECTIVE EXPERIMENT

  • Eleven photography experts labeled the benchmark in standard indoor lighting on a 4K monitor, annotating up to 30 images daily to limit fatigue.At least three other experts reviewed every annotation before acceptance.Source: §A.1.1 · p. 13

A.1.2 MORE DETAILS ON LLVISIONQA

  • LLVisionQA covers distortions including blur, noise, artifacts, exposure issues, and other artificial distortions, alongside color, lighting, composition, visual style, and photographic methods.Examples include gaussian blur, compression artifacts, under- or over-exposure, symmetry, animation, and background bokeh.Source: §A.1.2 · p. 13
  • LLVisionQA pairs global distortion questions with in-context versions about specific image regions, such as asking whether the whole image or a tree is blurred.It also contrasts global clarity questions with questions about the clarity of a person’s face.Source: §A.1.2 · p. 13
  • For other low-level attributes, LLVisionQA similarly contrasts global questions about color or brightness with in-context questions about a house’s color or the darkest object.Source: §A.1.2 · p. 13
  • The benchmark’s annotation design spans both LLVisionQA question-answer interfaces for perception and LLDescribe text-description interfaces for description.Source: Figure 5 · p. 14

A.2.1 EVALUATION DETAILS FOR PERCEPTION ABILITY

The perception evaluation addresses Kosmos-2’s failure to follow multiple-choice prompts with close-set inference and uses repeated GPT grading to improve evaluation reliability.

A.2.1 EVALUATION DETAILS FOR PERCEPTION ABILITY

  • In the primary-sample test, Kosmos-2 appended a new candidate answer instead of choosing among the options in 86% of situations; a simple prompt modification reduced failures to less than 10% in the sample set.Source: §A.2.1 · p. 14
  • Perplexity-based close-set inference notably improves Kosmos-2’s LLVisionQA results, supporting the observation that prompt failures impair its multiple-choice performance.The method ranks candidate answers by generative loss and selects the answer with the minimum loss; the paper describes Kosmos-2’s results as decent for its model size.Source: §A.2.1 · p. 15, Table 5 · p. 15
  • Five-round GPT voting raised evaluation accuracy from 93.2% to 98.4%, reducing errors to one-quarter of single-round evaluation errors.The strategy repeats the same prompt five times and selects the most popular answer.Source: §A.2.1 · p. 15

A.2.2 EVALUATION DETAILS FOR DESCRIPTION ABILITY

The description evaluation uses GPT to assess MLLM descriptions against LLDescribe golden descriptions on three low-level criteria, averaging five repeated GPT ratings to reduce variability.

A.2.2 EVALUATION DETAILS FOR DESCRIPTION ABILITY

  • The general MLLM prompt asks for detailed descriptions of image quality, aesthetics, and other low-level appearance.The evaluation templates use the MLLM output and LLDescribe golden description as the respective descriptions being compared.Source: §A.2.2 · pp. 16–17
  • The evaluation repeats each GPT prompt five times and averages the answers to mitigate GPT’s variability.The paper presents this pooling strategy as a way to obtain a more accurate final score.Source: §A.2.2 · p. 16
  • GPT rates completeness, precision, and relevance on 0–2 scales, evaluating information coverage, contradictory low-level claims, and relevance to low-level visual attributes.The criteria compare the MLLM output description with the LLDescribe golden description; relevance may cover blur, noise, exposure, artifacts, color, lighting, focus, or composition.Source: §A.2.2 · pp. 16–17

A.2.3 EVALUATION DETAILS FOR ASSESSMENT ABILITY · A.3 EXTENDED EXPERIMENTAL RESULTS · A.3.1 ARCHITECTURES OF DIFFERENT MLLMS

The evaluation appendix details MLLM quality-scoring procedures and dataset protocols, reports that MLLM-specific high/low tokens generally outperform good/poor tokens, and describes the evaluated models’ architectures.

A.2.3 EVALUATION DETAILS FOR ASSESSMENT ABILITY

  • The IQA evaluation uses a short, nine-line algorithm that can be integrated with causal-LLM-based MLLMs to predict quantitative image-quality scores.The appendix also aligns CLIP-ViT-Large-14 with MLLMs using 224 × 224 inputs and softmax pooling over cosine similarities to good- and poor-quality prompts.Source: §A.2.3 · p. 18
  • For InstructBLIP, the scoring substitutes high/low probabilities for good/poor because high and low are its most frequent output tokens.The passage reports frequencies of 89% for high and 8% for low.Source: §A.2.3 · p. 18
  • High/low token pairs generally predict IQA better than good/poor pairs on most databases, supporting use of MLLMs’ frequent output tokens.The authors report this result in their validation of the token-selection methodology.Source: §A.2.3 · p. 19
  • The synonym-ensemble strategy defines positive and negative token sets, including good, fine, high and poor, bad, low, respectively.The appendix directs readers to Table 11 for results across different positive and negative sets.Source: §A.2.3 · p. 19
  • For CGIQA-6K (Zhang et al., 2023b), the authors evaluate game and movie subsets separately and average their results, applying the same protocol to NIQE and CLIP-ViT-Large-14.Each subset contains 3,000 images and uses different instructions for human annotators.Source: §A.2.3 · p. 19

A.3.1 ARCHITECTURES OF DIFFERENT MLLMS

  • The 15 evaluated MLLM variants combine a CLIP version with a large language model, connected through projection layers, a Q-Former, or cross-attention.The variants differ in vision and language architectures and in how the modalities are aligned.Source: §A.3.1 · p. 20

A.3.2 EXTENDED RESULTS FOR PERCEPTION · “Yes or No?”: How Biased are MLLMs?

The extended results report similar MLLM performance on LLVisionQA’s dev and test subsets, alongside benefits from context and systematic yes/no bias in perception judgments.

A.3.2 EXTENDED RESULTS FOR PERCEPTION

  • MLLMs perform similarly on LLVisionQA’s dev and test subsets, supporting the benchmark’s reliability and stability for low-level question answering.The dev subset had not been seen by evaluated models and is intended as a cross-validation reference; its results are not yet public.Source: §A.3.2 · p. 20
  • Adding content context does not degrade MLLM perception; models can answer in-context questions better, suggesting suitable higher-level prompts may improve precision.Source: §A.3.2 · p. 20
  • MLLMs show strong ability on “what” questions, which the authors identify as suggesting potential reasoning abilities.Source: §A.3.2 · p. 20

“Yes or No?”: How Biased are MLLMs?

A.3.3 EXTENDED RESULTS FOR DESCRIPTION · A Qualitative Comparison on the Descriptions:

The extended results show that MLLMs remain unreliable at describing low-level image attributes, with perception—not instruction relevance—the main bottleneck. Qualitative examples reveal divergent descriptions, especially for AI-generated images, and longer outputs often include more irrelevant information.

A.3.3 EXTENDED RESULTS FOR DESCRIPTION

  • Current MLLMs struggle to provide complete and accurate low-level visual descriptions.This is the central finding of the extended description results.Source: §A.3.3 · p. 21
  • Relevance scores are generally higher, suggesting low-level attribute perception is the main bottleneck in improving description ability.The figure reports scores across three description dimensions, normalized to [0, 1].Source: §A.3.3 · p. 21, Figure 11 · p. 22

A Qualitative Comparison on the Descriptions:

A.3.4 EXTENDED RESULTS FOR ASSESSMENT · A Deep Dive Into the Probabilities: · A.3.5 Synonym Ensemble: FURTHER IMPROVING IQA ABILITY (A3) FOR MLLMS

The assessment experiments show that MLLMs can correlate with human quality judgments, especially on coarse cases, while the probability and synonym-ensemble studies identify ways to improve IQA predictions and remaining prompt sensitivity.

A.3.4 EXTENDED RESULTS FOR ASSESSMENT

  • Across seven IQA datasets, MLLMs positively correlate with human ratings and can align very highly on coarser cases such as AGIQA-3K and SPAQ.This occurs without direct alignment to human perception and with the simple prompt “Rate the quality of the image.”Source: §A.3.4 · pp. 22, 25, Figure 14 · p. 25
  • Their predictions remain inaccurate on finer-grained cases such as LIVE-FB and CGIQA-6K, indicating a need for well-designed fine-tuning to improve measurability.LIVE-FB contains more than 95% high-quality images; CGIQA-6K consists entirely of relatively high-quality images from video games or movies.Source: §A.3.4 · p. 25

A Deep Dive Into the Probabilities:

  • For every evaluated MLLM and IQA dataset, softmax predicts human ratings better than argmax, which reduces outputs to only 0 or 1.The comparison supports quantitative assessment ability beneath MLLMs’ word outputs and the effectiveness of the softmax-based IQA strategy.Source: A Deep Dive Into the Probabilities · p. 25
  • For InstructBLIP (Flan-T5-XL), high↔low tokens outperform good↔bad tokens in IQA, suggesting that top-frequency tokens are more quality-distinctive.The T5-backed model favors high↔low more than its Vicuna-7B-based counterpart despite the same pre-training settings.Source: A Deep Dive Into the Probabilities · p. 25, Table 10 · p. 25

A.3.5 Synonym Ensemble: FURTHER IMPROVING IQA ABILITY (A3) FOR MLLMS

  • The synonym ensemble improves top-five MLLMs’ IQA accuracy by up to 2% (1.3% on average), while their rankings remain unchanged.The top-five methods are InternLM-XComposer-VL, QWen-VL, LLaVA-v1.5 (13B), mPLUG-Owl, and LLaVA-v1.5 (7B).Source: §A.3.5 · p. 26, Table 11 · p. 26
  • Prompt combinations are model-dependent: good+fine↔poor+bad works best for InternLM-XComposer-VL but reduces QWen-VL accuracy versus good↔poor.Good↔poor is the overall best single word pair and produces stable results across MLLMs, so the authors retain it in Q-Bench.Source: §A.3.5 · p. 26

B STATEMENT ON DATA CONTAMINATION

Q-Bench limits contamination in its perception and description tasks by keeping evaluation data private, while IQA assessment contamination is less certain. The authors distinguish language-level IQA knowledge from image-dataset contamination because language models cannot see the images.

Data contamination

  • The perception and description datasets’ questions, answers, and descriptions were unseen by existing MLLMs, with the LLVisionQA test subset and all LLDescribe labels kept private.The authors intend this safeguard to preserve Q-Bench’s long-term value as an indicator of low-level visual abilities.Source: B STATEMENT ON DATA CONTAMINATION · p. 26
  • For tested open-source models, the authors know of no IQA dataset used in multimodal training, based on their technical reports.They describe IQA contamination as more complicated than for the two tasks evaluated on Q-Bench’s own datasets.Source: B STATEMENT ON DATA CONTAMINATION · p. 26
  • The authors do not regard language-stage IQA knowledge as contamination because language models cannot see images, and argue it helps MLLMs link visual attributes to human quality opinions.Their example is knowledge that a blurry image has low quality.Source: B STATEMENT ON DATA CONTAMINATION · p. 27

C LIMITATIONS AND DISCUSSIONS

Q-Bench’s limitations include an imbalanced LLVisionQA question mix, subjective and potentially hallucination-prone evaluation of descriptions, and no direct guidance for improving low-level abilities. The authors propose dataset balancing, more reliable evaluation, and scaling the datasets for future instruction tuning.

C LIMITATIONS AND DISCUSSIONS

  • LLVisionQA has 62% Yes-questions and 38% No-questions, which may bias MLLM comparisons; the authors plan reversed versions of questions to balance the dataset.The concern is notable because MLLMs frequently answer “yes” to Yes-or-No questions.Source: C LIMITATIONS AND DISCUSSIONS · p. 27
  • Description matching is subjective, and the 5-round GPT-assisted protocol may still hallucinate, motivating further work on a more reliable evaluation protocol.Human ratings also vary between individuals.Source: C LIMITATIONS AND DISCUSSIONS · p. 27
  • Q-Bench does not directly guide capability enhancement; the authors plan to scale LLDescribe and LLVisionQA toward reliable low-level visual instruction tuning.They intend this tuning process to improve MLLMs’ low-level abilities.Source: C LIMITATIONS AND DISCUSSIONS · p. 27

Related papers

Loading 2309.14181v3…