PaperlayerView exact revision on arXiv

Source-linked AI summary

Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models

Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Kaixin Xu, Chunyi Li, Jingwen Hou, Guangtao Zhai, Geng Xue, Wenxiu Sun, Qiong Yan, Weisi Lin

arXiv:2311.06783v1cs.CVcs.MM

TL;DR

Existing MLLMs have unsatisfactory accuracy on low-level visual queries, while available training data largely emphasizes high-level vision. The paper collects human pathway feedbacks and converts them into 200K diverse instruction-response pairs, and reports consistent low-level capability improvements across models, including a +0.243 average SRCC gain on never-seen datasets. The authors also report trade-offs and remaining gaps in general-task performance, human-level accuracy, and image-domain coverage.

  • Problem

    Existing MLLMs respond to low-level visual queries with unsatisfactory accuracy, while publicly available training datasets generally focus on high-level visual abilities.

  • Method

    The paper collects human pathway feedbacks describing low-level attributes and overall quality, then uses GPT to convert them into diverse instruction-response pairs for MLLM tuning.

  • Results

    +0.243 average SRCC gain on three never-seen datasets demonstrates robust low-level perception improvements across a broad domain.

  • Takeaways & Limitations

    Q-Instruct improves low-level visual question answering and perception, while text-driven tuning can align MLLMs with numerical quality scores and generalize to unseen visual inputs.

  • Takeaways & Limitations

    Q-Instruct-tuned models show declined general-purpose performance, remain below average human accuracy at 68%-71% versus about 74%, and mainly use natural in-the-wild images.

Abstract

from arXiv · show

Multi-modality foundation models, as represented by GPT-4V, have brought a new paradigm for low-level visual perception and understanding tasks, that can respond to a broad range of natural human instructions in a model. While existing foundation models have shown exciting potentials on low-level visual tasks, their related abilities are still preliminary and need to be improved. In order to enhance these models, we conduct a large-scale subjective experiment collecting a vast number of real human feedbacks on low-level vision. Each feedback follows a pathway that starts with a detailed description on the low-level visual appearance (*e.g. clarity, color, brightness* of an image, and ends with an overall conclusion, with an average length of 45 words. The constructed **Q-Pathway** dataset includes 58K detailed human feedbacks on 18,973 images with diverse low-level appearance. Moreover, to enable foundation models to robustly respond to diverse types of questions, we design a GPT-participated conversion to process these feedbacks into diverse-format 200K instruction-response pairs. Experimental results indicate that the **Q-Instruct** consistently elevates low-level perception and understanding abilities across several foundational models. We anticipate that our datasets can pave the way for a future that general intelligence can perceive, understand low-level visual appearance and evaluate visual quality like a human. Our dataset, model zoo, and demo is published at: https://q-future.github.io/Q-Instruct.

1. Introduction

Existing multimodal language models show potential for low-level visual understanding, but their accuracy remains unsatisfactory because training datasets largely emphasize high-level vision. Q-Instruct addresses this gap by converting human low-level visual feedback into diverse instruction data for tuning.

  • Existing MLLMs can answer low-level visual queries, but their response accuracy remains unsatisfactory.
  • The primary data gap is that publicly available MLLM training datasets generally focus on high-level visual abilities.
  • Q-Pathway collects direct human feedback describing low-level attributes and overall visual quality.
  • GPT converts pathway feedbacks into diverse question-answer pairs and extended conversations for broader low-level visual instruction tuning.
  • The resulting Q-Instruct contains 200K instruction-response pairs and improves various low-level abilities across multiple MLLM base models.

2. Related Works

Prior work covers image quality assessment and high-level multimodal capabilities, while Q-Pathway is positioned against these sources as a low-level visual dataset with a more balanced sampled distribution.

  • Image quality assessment datasets target scores aligned with integrated human opinions across low-level visual aspects.
  • Existing IQA datasets address artificially distorted images, in-the-wild photographs, and AI-generated content.
  • Q-Pathway subsamples source images to reduce MOS-distribution skew and produce a more balanced sampled distribution.
  • Multimodal language models extend language models with visual inputs for high-level tasks such as captioning and visual question answering.

3. the Q-Pathway

Q-Pathway is a large-scale human-feedback dataset built from diverse and distribution-balanced images. Its pathway annotations describe contextual low-level attributes before giving an overall quality conclusion, with feedbacks showing consistency across annotators.

  • 3.1. Preparation of Images: Q-Pathway collects human text feedback on low-level visual perception and quality assessment from images sampled across multiple sources.
  • 3.1. Preparation of Images: The assembled image set contains 18,973 images, including artificially corrupted COCO images and content from in-the-wild and AI-generated sources.
  • 3.2. Task Definition: the pathway Feedbacks: Each pathway feedback exhaustively describes contextual low-level attributes such as noise, brightness, and clarity, then provides a general conclusion.
  • 3.2. Task Definition: the pathway Feedbacks: The pathway format preserves positional context and follows the ordering of human reasoning from observed attributes to overall quality.
  • 3.4. Analysis: Feedbacks from different subjects for the same image show decent consistency and complement one another.
  • 3.4. Analysis: Feedbacks generally range from 20 to 100 words, average 46.4 words, and contain positive and negative feedbacks in similar proportions.

4. the Q-Instruct

Q-Instruct expands Q-Pathway feedbacks into multiple instruction formats so MLLMs can describe, answer questions about, and discuss low-level visual properties. The dataset combines pathway reasoning, VQA, and extended conversations generated with GPT assistance.

  • Q-Instruct is designed to activate varied low-level visual abilities and support responses to diverse human queries.
  • The 58K pathway feedbacks form a primary Q-Instruct subset for low-level reasoning and quality prediction.
  • GPT converts pathway feedbacks into 76K diverse VQA questions with concise attribute-, opinion-, and context-related answers.
  • Extended conversations address causes of visual patterns, photography improvements, image editing, recommendations, and related discussions.
  • The evaluated training strategies either mix Q-Instruct with high-level instruction data or apply Q-Instruct in a further low-level tuning stage.

5. Low-level Visual Instruction Tuning

The paper evaluates when to introduce Q-Instruct during MLLM training, comparing mixing it with high-level data against a separate post-tuning stage. Both strategies improve low-level visual tasks, with high-level awareness contributing to their effectiveness.

  • Training Strategies: Q-Instruct can be mixed with high-level datasets during second-stage tuning or used alone in a third stage afterward.These are the two training strategies evaluated for incorporating low-level visual instruction data.
  • Training Strategies: Both Q-Instruct training strategies produce notable improvements across various low-level visual tasks.The comparison covers mixing Q-Instruct with high-level datasets and using it after original high-level tuning.
  • Training Strategies: High-level awareness contributes to the effectiveness of both Q-Instruct training strategies.The authors retain or involve high-level instruction data when evaluating the two approaches.

6. Experiments

Experiments evaluate Q-Instruct across multiple MLLMs and low-level perception, description, and quality-assessment tasks. The results show improvements across seen and unseen domains, continued gains with more data, benefits from joint training, and the importance of retaining high-level awareness.

  • Experimental Setups: Four variants of three state-of-the-art MLLMs are evaluated before and after Q-Instruct training under both proposed strategies.The original high-level datasets remain unchanged across the comparisons.
  • Experimental Setups: The evaluation covers Perception, Description, and Image Quality Assessment, measuring low-level question answering, textual transformation, and quality scores.Perception uses MCQ accuracy, while IQA extracts quality scores from MLLMs through softmax pooling.
  • Main Results: Q-Instruct improves IQA performance on mostly seen, barely seen, and never seen datasets without using MOS values during training.The reported results indicate generalization across datasets with different degrees of overlap and domain differences.
  • Main Results: +0.243 average SRCC gain is achieved on three never-seen IQA datasets spanning computer-generated images, artificially degraded images, and videos.These datasets have a notable domain gap from the mostly in-the-wild photographs in Q-Instruct.
  • Ablation Studies: Scaling Q-Instruct continuously improves low-level perceptual accuracy, with performance not saturated at the current 200K data scale.This pattern holds under both the mix and after training strategies.
  • Ablation Studies: Jointly training the Q-Instruct subsets improves low-level description and question-answering accuracy, especially when low-level data is used independently.The subsets originate from the same human feedbacks, making the improvement cost-efficient according to the authors.
  • Ablation Studies: Replacing high-level instruction data with Q-Instruct performs notably worse than the two adopted strategies, indicating the importance of high-level awareness.The comparison is made on low-level Perception ability using the test set.

7. Conclusion

The paper introduces Q-Pathway and Q-Instruct datasets for low-level visual instruction tuning and reports improvements in perception, description, and IQA. It also reports generalization to unseen visual inputs from text-only supervision of quality scores.

  • Conclusion: Q-Pathway contains 58K human text feedbacks, while derived Q-Instruct contains 200K instruction-response pairs for low-level visual tuning.The datasets are designed to facilitate low-level visual instruction tuning for MLLMs.
  • Conclusion: Q-Instruct improves question-answering accuracy for low-level visual perception and supports more reliable image descriptions.The conclusion presents these as the main low-level visual capability gains.
  • Conclusion: Text-driven instruction tuning aligns MLLMs with numerical quality scores and generalizes to unseen types of visual inputs.The conclusion describes this result as applying to Image Quality Assessment.

A.1. Interface for Subjective Experiments

The subjective-experiment interface records participant feedback for individual images while displaying MOS and low-level attributes as references. GPT prompts convert image-quality descriptions into concise questions, answers, and false candidates.

  • Interface for Subjective Experiments: Participants record their IDs and write pathway feedback for a given image in the local Gradio interface.The interface was built with Gradio 3.34.0 on Ubuntu 20.04 workstations.
  • Interface for Subjective Experiments: The interface displays the image MOS and possible low-level attributes as references during feedback collection.A screenshot of the interface is provided in Figure 7.
  • Question Generation: GPT generates concise What, Why, or How questions, core answers, and reasonable false candidates from image-quality descriptions.The generated outputs are organized as a JSON list, with false answers contradicting the description.

B. Hyper-parameters during Training

The study specifies model- and task-specific training settings, while evaluating low-level perception, description, and quality assessment through standardized prompts and demonstrations.

  • Training resources: LLaVA-v1.5 low-level instruction tuning used 8 NVIDIA A100-SMX4-80GB GPUs, requiring 16 hours for 7B and 22 hours for 13B mix training.The corresponding hyper-parameters are recorded in Table 10.
  • Training resources: mPLUG-Owl-2 and InternLM-XComposer-VL mix training each used 32 NVIDIA A100-SMX4-80GB GPUs, requiring 8 and 13 hours, respectively.Their hyper-parameters are listed in Tables 11 and 12.
  • A1 Perception: For A1 Perception, model-specific prompts represent images, questions, and choices, with training VQA data transformed under the same settings.LLaVA-v1.5 answers with the option letter directly, while InternLM-XComposer-VL uses a model-specific answer template.
  • Qualitative analysis: Qualitative demonstrations cover multi-turn rating and reasoning, improvement suggestions, multi-purpose and multi-format evaluation, correction, and local in-context perception.The correction example shows rating modification after additional context, while local analysis distinguishes bright and dark regions or object clarity.
  • A2 Description: For A2 Description, all models use “Describe and evaluate the quality of the image.” to elicit low-level attributes and image-quality evaluation.The unified prompt was selected because it effectively supports description across all base models.
  • A3 Quality Assessment: For A3 Quality Assessment, softmax output between good and poor provides quantifiable image scores from the first output token.For KoNViD-1k, one frame per second is sampled and frame scores are averaged, although the authors acknowledge this is not optimal for video quality assessment.

E. Limitations

The authors report trade-offs and scope boundaries: tuning improves low-level abilities but can reduce general-purpose performance, remains below average human accuracy, and is centered on natural in-the-wild images.

  • Performance trade-offs: Q-Instruct-tuned models show declined performance on general-purpose tasks, especially language-centric and heavy-reasoning tasks.The authors warn that such models may produce unwanted outputs outside low-level visual perception and understanding.
  • Human comparison: Q-Instruct-tuned models reach 68%-71% accuracy on LLVisionQA-test, below the average human performance of about 74%.The authors state that the models may not yet directly replace humans on low-level related tasks.
  • Data scope: The Q-Instruct dataset mainly contains natural in-the-wild images, although the models generalize well to other visual-content types.The authors note that further tuning on those datasets might improve performance.
Loading 2311.06783v1…