Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models

1. Introduction

• The authors seek a unified foundation model that handles open-ended queries across image quality assessment, low-level attributes, and visual dimensions.Examples include identifying noise or blur and evaluating clarity or brightness.Source: §1 · p. 1
• The paper identifies inadequate low-level visual accuracy and a lack of training datasets focused on these abilities as problems for existing MLLMs.Publicly available datasets generally focus on high-level visual abilities.Source: §1 · p. 2
• Q-Pathway contains 58K human feedbacks on 18,973 images, each with at least three feedbacks and an average length of 46.4 words.Each feedback describes low-level attributes with content or position context, then gives an overall image-quality conclusion, capturing human perception and reasoning.Source: §1 · p. 2
• GPT-assisted conversion turns pathway feedbacks into Q-Instruct’s 200K instruction-response pairs, including VQA, balanced yes-or-no questions, and long conversations.VQA examples use multiple-choice or direct-answer formats and cover attributes, image context, and quality; conversations address low-level concerns such as distortions and improving image quality.Source: §1 · p. 2
• Q-Instruct is presented as improving diverse low-level visual abilities in MLLMs through low-level visual instruction tuning.The authors report experiments validating this improvement and say they offer insights for bringing low-level abilities into general foundation models.Source: §1 · p. 2

2.1. Low-level Visual Perception

• Existing IQA datasets target artificial distortions, in-the-wild photos [14], and AI-generated content, while newer tasks cover visual dimensions [9, 56] and distortions.Low-level perception encompasses human-perceived image appearance beyond object-level content; relating these tasks in datasets could help one foundation model learn them together.Source: §2.1 · p. 3
• IQA approaches range from training-free metrics and task-specific deep models [18] to CLIP-based methods.Traditional metrics are robust but less accurate, deep methods can generalize poorly, and multimodal methods are robust across scenarios but not sufficiently accurate case by case; finer-grained abilities and cross-task reasoning remain underexplored. Q-Instruct instruction tuning improves MLLM low-level abilities.Source: §2.1 · p. 3

2.2. Multi-modality Large Language Models

• MLLMs combine visual inputs with language models to support high-level tasks including image captioning [1] and visual question answering [32] [40].A recent benchmark finds their low-level visual abilities unsatisfactory, especially on fine-grained questions; the authors attribute this mainly to missing data and introduce Q-Instruct as a low-level visual instruction-tuning dataset.Source: §2.2 · p. 3

3. the Q-Pathway

• Q-Pathway is presented as a large-scale dataset of human feedback on low-level visual aspects, with image sampling, pathway task definition, an in-lab study, and dataset analysis as its construction components.The authors describe reducing skews in source distributions and training subjects before they provide feedback.Source: §3 · p. 3

3.1. Preparation of Images

• The 18,973-image set draws from diverse sources, balances positive and negative appearances, and includes 1,012 pristine COCO images randomly altered with one of 15 distortions.The source pool includes four in-the-wild IQA datasets and two AI-generated-image datasets.Source: §3.1 · pp. 3–4

3.2. Task Definition: the pathway Feedbacks

• Pathway feedbacks pair an exhaustive description of low-level attributes with a general conclusion, preserving positional context and aligning annotation order with human reasoning.The format is intended to help multimodal language models emulate human perception and understanding of low-level vision.Source: §3.2 · p. 4

3.3. The subjective study process.

• In a controlled laboratory study, 39 trained subjects provided 58K pathway feedbacks after calibration on quality and low-level attributes.MOSs were shown for most images; the study warned subjects after more than 30 consecutive images and paused the task after 50.Source: §3.3 · p. 4

3.4. Analysis

• Feedbacks averaged 46.4 words, generally ranged from 20 to 100 words, preserved attribute context, showed decent inter-subject consistency, and covered diverse attributes with balanced positive and negative feedback.The reported average is about four times the length of common high-level image captions.Source: §3.4 · pp. 4–5

4.1. Low-level Reasoning with pathway Feedbacks

• Pathway feedbacks describe low-level attributes such as brightness and clarity through to overall quality, with 58K examples used for reasoning-focused instruction tuning.Source: §4.1 · p. 5

4.2. Visual Question Answering (VQA)

• The GPT-assisted VQA subset contains 76K diverse how/what questions, 57K balanced yes/no questions, and multiple-choice versions with distracting answers.Source: §4.2 · p. 5

4.3. Extended Conversations

• The 12K GPT-generated extended conversations cover visual-pattern causes, photography suggestions, image restoration, enhancement or editing, consumer recommendations, and other discussions grounded in pathway descriptions.Source: §4.3 · pp. 5–6

• Figure 5 compares mixing Q-Instruct with high-level visual instruction data against adding a low-level-only tuning stage after high-level tuning.Source: Figure 5 · p. 6

Low-level Visual Instruction Tuning

• Open-source MLLM training typically aligns visual and language representations before visual instruction tuning with human-labeled datasets [24].The alignment stage uses million-scale web data.Source: §5 · p. 6
• Q-Instruct can be mixed with high-level datasets during the second-stage visual instruction tuning, or used alone in a third stage after high-level tuning.The mixed-data strategy aims to build low-level abilities within general high-level awareness.Source: §5 · p. 6
• Both Q-Instruct training strategies bring notable improvements across various low-level visual tasks, with high-level awareness contributing to their effectiveness.Source: §5 · p. 6

6.1. Experimental Setups

• Experiments evaluate four variants of three state-of-the-art MLLMs under both strategies, using default tuning hyperparameters, final checkpoints, and image-overlap filtering.The original high-level dataset combination remains unchanged during evaluation.Source: §6.1 · p. 6

6.2. Main Results

• Both training strategies significantly improve low-level MCQ perception; Yes-or-No accuracy gains average more than 10%, with larger improvements on distortions than other attributes.Source: §6.2 · p. 6, Table 3 · p. 7, Table 4 · p. 7
• Tuning improves low-level descriptions across variants (>1.5/2 average score), especially relevance (+0.31), versus smaller completeness (+0.17) and precision (+0.04) gains.The authors suggest captioning-like instructions may be insufficient for description tasks requiring longer responses.Source: §6.2 · p. 6, Table 5 · p. 7
• On three never-seen IQA datasets, low-level instruction tuning yields a +0.243 average SRCC gain, demonstrating robust generalization across domains.The study also reports strong performance on mostly seen datasets without using MOS values as training supervision.Source: §6.2 · p. 7, Table 6 · p. 8

6.3. Ablation Studies

• Scaling Q-Instruct continuously improves low-level perceptual accuracy under either strategy, and performance remains unsaturated at the current 200K-data scale.The scaling comparison is shown in Figure 6.Source: §6.3 · pp. 7–8, Figure 6 · p. 7
• Joint training improves low-level description and question-answering accuracy over task-separate tuning, especially when low-level data is used independently.The authors characterize this improvement as cost-efficient because the Q-Instruct subsets derive from the same human feedbacks.Source: §6.3 · p. 8, Table 7 · p. 8, Table 8 · p. 8
• Replacing high-level tuning with low-level data performs notably worse than the two adopted strategies, indicating high-level awareness matters for general low-level visual recognition.Source: §6.3 · p. 8, Table 9 · p. 8

• Q-Pathway’s 58K human text feedbacks and derived Q-Instruct’s 200K instruction-response pairs facilitate low-level visual instruction tuning and significantly improve MLLM question-answering accuracy on low-level perception.Source: §7 · p. 8
• Pure text-driven instruction tuning can sufficiently align MLLMs with numerical quality scores, with impressive generalization to unseen types of visual inputs.Source: §7 · p. 8
• The datasets showcase potential for more reliable low-level image descriptions and eventually relieving humans of this task.Source: §7 · p. 8

A.1. Interface for Subjective Experiments

• The subjective-experiment interface, built with Gradio 3.34.0 on Ubuntu 20.04 workstations, records participant IDs and pathway feedback, with image MOS and possible low-level attributes as references.The paper also provides a screenshot of the interface in Fig. 7.Source: §A.1 · p. 12

A.2. Prompts for Building Q-Instruct with GPT

• What/How prompts request concise core answers and reasonable false candidates that contradict the image-quality description.The requested output is a JSON list.Source: §A.2 · p. 12
• Yes/No prompts request concise binary answers, roughly balanced between Yes and No, and assign No to questions about unmentioned quality issues.The requested output is a JSON list.Source: §A.2 · p. 12
• Extended-conversation prompts cover visual-pattern causes, photography improvements, image-restoration or editing tools, consumer recommendations, and other relevant topics.The prompts request query-response pairs in a JSON list.Source: §A.2 · p. 12

B. Hyper-parameters during Training

• LLaVA-v1.5 tuning used 8 A100s (16 hours for 7B, 22 for 13B); mPLUG-Owl-2 and InternLM-XComposer-VL used 32 A100s (8 and 13 hours, respectively).The reported durations are for the mix version; the hyperparameters are stated to match those of the original models.Source: B. Hyper-parameters during Training · p. 12, Table 10 · p. 12, Table 11 · p. 13

C.1. Prompt Settings on (A1) Perception (via MCQ)

• MCQ prompts differ across models but request a choice letter directly, and the VQA subset is transformed using each model’s corresponding settings.InternLM-XComposer-VL’s template includes “Context: N/A” and presents A/B/C options; the LLaVA-v1.5 and mPLUG-Owl-2 templates also request the option letter.Source: §C.1 · pp. 12–13

D. Qualitative Analyses

• The demo illustrates multi-turn quality rating, reasoning, improvement suggestions, and prioritization of the most important suggestion.The authors report inviting team members to chat with the publicly accessible Q-Instruct-mPLUG-Owl-2 demo under the mix strategy.Source: Figure 8 · p. 14, D. Qualitative Analyses · p. 14

C.2. Prompt Setting on (A2) Description

• For (A2), all models receive “Describe and evaluate the quality of the image.” to elicit low-level attributes and image-quality evaluation.The alternative “Rate the quality of the image. Think step by step.” prompt led InternLM-XComposer-VL to provide only a number, without explanation or reasoning.Source: §C.2 · p. 14

C.3. Prompt Setting on (A3) Assessment

• On KoNViD-1k, scoring one frame per second and averaging still significantly outperformed methods not trained for video quality assessment, despite being a nonoptimal strategy.For (A3), scores are collected from the softmax output between “good” and “poor” under the first output token of the MLLM.Source: §C.3 · p. 14

D. Qualitative Analyses

• The demo also shows multi-purpose, multi-format quality evaluation and correction of a rating when the model receives context that background bokeh is intentional.The authors additionally report local in-context perception among the demo’s capabilities.Source: D. Qualitative Analyses · p. 14, Figure 9 · p. 15, Figure 10 · p. 15

E. Limitations

• Q-Instruct-tuned models show declined performance on general-purpose tasks, especially language-centric or reasoning-heavy tasks.The authors warn they may produce unwanted outputs when used outside low-level visual perception and understanding.Source: E. Limitations · p. 15
• Models may produce unwanted outputs when applied beyond low-level visual perception and understanding.Source: E. Limitations · p. 15
• Q-Instruct-tuned models reach 68%-71% accuracy on LLVisionQA-test, below the average human’s about 74%.The authors say the models may not yet directly replace humans on low-level tasks.Source: E. Limitations · p. 15
• The dataset mainly contains natural in-the-wild images, and further tuning on other visual contents could improve performance.Source: E. Limitations · p. 15

F. Ethical Acknowledgements

• Participants were informed about the nature and amount of tasks beforehand, and none reported uncomfortable content.The authors express gratitude and commit to participant well-being and research integrity.Source: F. Ethical Acknowledgements · p. 16

H. License

• Researchers and open-source developers may use the Q-Instruct dataset and fine-tuned weights for four MLLMs; commercial use requires prior permission and compliance with base-model licenses.Source: H. License · p. 16