PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Open-ended Visual Quality Comparison

Haoning Wu, Hanwei Zhu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Annan Wang, Wenxiu Sun, Qiong Yan, Xiaohong Liu, Guangtao Zhai, Shiqi Wang, Weisi Lin

arXiv:2402.16641v2cs.CV

TL;DR

Existing comparative datasets and methods generally lack open-ended quality comparison, and open-source LMMs trained on single-image instructions lack capability in image comparison. Merge2Compare constructs comparative data through pair/group matching, top-similarity pair removal, and LLM merging, which converts single-image evaluations into comparative texts. Co-Instruct outperforms existing LMMs, including GPT-4V, on visual quality comparison and is reported to surpass human accuracy on related settings.

  • Problem

    Existing comparative datasets and methods generally lack open-ended quality comparison, and open-source LMMs trained on single-image instructions lack capability in image comparison.

    Source: §1 · p. 2, §7 · p. 14

  • Method

    Merge2Compare constructs comparative data through pair/group matching, top-similarity pair removal, and LLM merging, which converts single-image evaluations into comparative texts.

    Source: §3.1 · pp. 5–6

  • Results

    Co-Instruct outperforms existing LMMs, including GPT-4V, on visual quality comparison and is reported to surpass human accuracy on related settings.

    Source: §7 · p. 14

  • Takeaways & Limitations

    The authors present open weights as allowing broader application than proprietary models.

    Source: §1 · p. 4

Abstract

from arXiv · show

Comparative settings (e.g. pairwise choice, listwise ranking) have been adopted by a wide range of subjective studies for image quality assessment (IQA), as it inherently standardizes the evaluation criteria across different observers and offer more clear-cut responses. In this work, we extend the edge of emerging large multi-modality models (LMMs) to further advance visual quality comparison into open-ended settings, that 1) can respond to open-range questions on quality comparison; 2) can provide detailed reasonings beyond direct answers. To this end, we propose the Co-Instruct. To train this first-of-its-kind open-source open-ended visual quality comparer, we collect the Co-Instruct-562K dataset, from two sources: (a) LLM-merged single image quality description, (b) GPT-4V "teacher" responses on unlabeled data. Furthermore, to better evaluate this setting, we propose the MICBench, the first benchmark on multi-image comparison for LMMs. We demonstrate that Co-Instruct not only achieves in average 30% higher accuracy than state-of-the-art open-source LMMs, but also outperforms GPT-4V (its teacher), on both existing related benchmarks and the proposed MICBench. Our model is published at https://huggingface.co/q-future/co-instruct.

1 Introduction

The introduction argues that open-ended visual quality comparison addresses ambiguity in absolute image evaluations, but existing comparison data and open-source LMM training do not support this setting well. Co-Instruct combines two weak-supervision data sources with a multi-image input design, and the paper reports strong comparisons against existing models while introducing MICBench.

1 Introduction

  • Comparative evaluations can yield agreement on image rankings despite disagreement on absolute exposure ratings, while existing datasets and methods lack open-ended comparison.The paper also identifies costly human collection and limited comparative ability in open-source LMMs trained on single-image instructions as gaps.Source: §1 · p. 2
  • Co-Instruct-562K combines LLM-merged human-labeled single-image descriptions with GPT-4V comparisons on unlabeled image groups.Merge2Compare starts from descriptions for 19K images; Teach2Compare uses 9K unlabeled images matched into 30K groups.Source: §1 · p. 3, Figure 3 · p. 3
  • Explicit image-text labels identify each image, while a visual abstractor reduces visual tokens from 1,025 to 65 per image before language decoding.The paper says linearly projecting visual embeddings for multiple images would exceed the language model’s context window and cause errors.Source: §1 · p. 3
  • Co-Instruct achieves up to 86% improvements over its baseline and is 61% better than the state-of-the-art open-source LMM.The introduction also reports that it surpasses GPT-4V on multiple-choice benchmarks and matches its detailed-reasoning ability.Source: §1 · p. 3
  • MICBench introduces 2,000 multiple-choice questions comparing quality or related attributes across groups of three or four images.Over half are Which questions; the benchmark also includes Yes-or-No and What/How/How-Many question types.Source: §1 · p. 4
  • Co-Instruct is released with open weights, which the authors say allows broader application than proprietary models.The contribution is presented alongside the dataset and MICBench as part of the paper’s three contributions.Source: §1 · p. 4

2 Related Works

Prior work uses comparative judgments for visual quality, but image-pair scaling and limited feedback constrain existing approaches. LMM quality-evaluation work largely focuses on single images, motivating instruction tuning for comparison.

2.1 Visual Quality Comparison

  • Pairwise comparisons grow exponentially with image count, while active-sampling methods are computationally expensive and unsuitable for large-scale experiments.Ranking approaches commonly output scalar scores or binary overall judgments, limiting feedback about specific distortions or preferences.Source: §2.1 · p. 4

2.2 LMMs for Visual Quality Evaluation

  • Earlier LMM quality-evaluation benchmarks and instruction-tuning work focus on single images, while open-source models trained on single-image data perform poorly on comparison settings.Co-Instruct-562K is introduced as instruction tuning for visual quality comparison.Source: §2.2 · p. 5
  • Table 1 reports Co-Instruct-562K dataset statistics, with # denoting “the number of”.Source: Table 1 · p. 5

3 Data Construction

Co-Instruct-562K is assembled without additional human annotation from two complementary sources: LLM-merged single-image descriptions and GPT-4V comparisons and question-answer pairs. Similarity filtering helps screen Merge2Compare data, while Teach2Compare adds diverse image comparisons and open-range Q&A.

3 Data Construction

  • Co-Instruct-562K avoids additional human annotation by converting existing image-quality descriptions and leveraging model abilities to build complementary instruction-tuning subsets.The construction follows three principles: convert existing information, learn from models, and collect complementary subsets.Source: §3 · p. 5

3.1 Merge2Compare.

  • Merge2Compare matches described images into pairs or groups, removes groups with high-similarity description pairs, then prompts an LLM to turn descriptions into comparative text.The LLM is asked to compare quality and explain why, or to rank groups of three or four images and justify the ranking.Source: §3.1 · p. 6, Figure 4 · p. 6
  • 96% of sampled Merge2Compare comparisons were correct, versus 72% among removed groups, supporting the top-similarity removal step.The human examination covered 250 random samples from each group.Source: §3.1 · p. 6

3.2 Teach2Compare.

  • Teach2Compare collects 9K diverse images, groups them into 30K combinations, and uses GPT-4V to produce general comparisons and quality-related question-answer pairs.A 250-sample spot check found around 94% correctness for Teach2Compare-general; examiners reported more content information than in Merge2Compare.Source: Figure 5 · p. 7, §3.2 · p. 7
  • Teach2Compare-Q&A provides 230K question-answer pairs from 30K image groups, converted into direct-answer and multiple-choice formats for training.GPT-4V generates questions and correct and false answers on reference aspects such as clarity, lighting, and color; failed generations are removed.Source: §3.2 · p. 7

3.3 Rationale of Combinations.

  • The subsets are intended to complement one another: Merge2Compare is more accurate but omits fine-grained cases, while Teach2Compare-general adds diverse scenarios and content information.The paper reports that joint training contributes to more comprehensive quality comparison, and that Teach2Compare-Q&A significantly enhances open-range answering ability.Source: §3.3 · p. 8

4 The Co-Instruct Model

Co-Instruct adapts its visual-token processing and image-text input format for multi-image comparison. The authors report that the interleaved format improves performance, including over learnable special tokens used to divide images.

4 The Co-Instruct Model

  • A visual abstractor reduces token counts before the language decoder, addressing context-window limits that make multi-image inputs impractical with large visual token sequences.The paper gives examples where two images exceed LLaVA's 2,048-token limit and four exceed LLaMA-2's 4,096-token window.Source: §4 · p. 8
  • The interleaved format labels each image in text before the query, rather than piling images before text, to help distinguish image identities during comparison.Source: §4 · p. 8
  • The image-text interleaved format significantly enhances Co-Instruct performance and is better than learnable special tokens for dividing images.Source: §4 · p. 8

5 The MICBench

MICBench evaluates open-range quality comparison among groups of three or four images. It combines diverse image sources with expert-crafted multiple-choice questions and separate development and test sets.

5 The MICBench

  • MICBench contains 2,000 groups of three or four images sampled from LLVisionQA and unlabeled databases, with zero overlap between the latter images and training images.The authors describe the resulting benchmark as covering varied quality issues and low-level appearances.Source: §5 · p. 9
  • Ten experts annotate MICBench questions, and a second expert cross-checks each answer; the benchmark has a 1,004-question dev set and a 996-question test set.Dev answers are public, while test answers are hidden.Source: §5 · p. 9
  • MICBench uses multiple-choice questions comparing quality or related attributes, including the special “Which” question type, to assess comparisons beyond two images.Source: §5 · p. 9

6 Evaluation

Co-Instruct is evaluated against open-source and proprietary LMMs on existing comparison benchmarks, single-image quality questions, and the new MICBench. It outperforms GPT-4V and human participants on MICBench, while ablations examine data composition, scale, and input format.

6.1 Implementation Details

  • The final checkpoint is evaluated on non-overlapping test subsets, with development subsets used to select training hyperparameters.Training takes 25 hours on eight NVIDIA A100 GPUs.Source: §6.1 · p. 10

6.2 Baseline Models

  • The evaluation compares Co-Instruct with five recent open-source LMMs supporting multi-image inputs and three proprietary models: Qwen-VL-Max, Gemini-Pro, and GPT-4V, Co-Instruct’s teacher.Co-Instruct is fine-tuned from the mPLUG-Owl2 checkpoint for two epochs at batch size 192; training took 25 hours on eight NVIDIA A100 GPUs.Source: §6.2 · p. 10, §6.1 · p. 10

6.3 Results on Existing Evaluation Settings

  • On Q-BenchPAIR-A1, Co-Instruct is 64% better than mPLUG-Owl2, 23% better than the best open-source baseline, and 2.7% better than GPT-4V.It is also the only LMM reported to surpass non-expert human accuracy on this benchmark.Source: Table 2 · p. 10, §6.3 · pp. 10–11
  • On Q-BenchPAIR-A2, comparative data improve Completeness by 57% and Precision by 59% over the version without comparative data, while Co-Instruct’s reasoning capability matches GPT-4V.The authors note GPT-4V outputs are more than twice as long, which may help explain its higher quantitative scores despite some incorrect information.Source: Table 3 · p. 11, §6.3 · pp. 11–12
  • On 2AFC-LMM, Co-Instruct outperforms all evaluated models, including GPT-4V, and has high consistency when image order is swapped.It is inferior on MM21 under aggregated scores, although its direct comparison accuracy there is 55.2% versus GPT-4V’s 54.4%.Source: Table 4 · p. 12, §6.3 · p. 12
  • On Q-BenchSINGLE-A1, comparative training improves Co-Instruct by 5% over single-image-only training and places it 4% ahead of GPT-4V.The authors report that comparative tuning and single-image quality evaluation can improve together in one model.Source: Table 5 · p. 12, §6.3 · p. 13

6.4 Results on MICBench

  • Co-Instruct is 5.7% more accurate than GPT-4V and 6.4% more accurate than non-expert human on MICBench’s three- and four-image comparisons.Open-source LMMs struggle to reach 50% accuracy; extending some models’ context windows for four images coincides with worse accuracy than on three-image groups.Source: Table 6 · p. 13, §6.4 · p. 13

6.5 Ablation Studies

  • Data ablations find that Merge2Compare improves capability across settings, Teach2Compare alone does not surpass GPT-4V, and heterogeneous subsets benefit question-answering results.Scaling general comparison data improves detailed reasoning, mixed data works better than homogeneous data at equal scale, and scaling Q&A data improves multiple MCQ metrics; the interleaved format also has non-negligible advantages.Source: §6.5 · pp. 13–14, Figure 9 · p. 14, Table 8 · p. 14

7 Conclusion

The paper introduces Co-Instruct, trained on Co-Instruct-562K, for open-ended visual quality comparison, and MICBench for evaluating comparisons across three or four images. It reports that Co-Instruct outperforms existing LMMs, including GPT-4V, and surpasses human accuracy on related settings.

7 Conclusion

  • Co-Instruct-562K combines LLM-merged human annotations on single images with GPT-4V responses, and trains Co-Instruct for open-ended quality comparisons with detailed reasoning.MICBench is introduced as the first LMM benchmark for multi-image quality comparison across three and four images.Source: §7 · p. 14

Related papers

Loading 2402.16641v2…