PaperlayerView exact revision on arXiv

Source-linked AI summary

Silkie: Preference Distillation for Large Visual Language Models

Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong

arXiv:2312.10665v1cs.CVcs.CL

TL;DR

Open LVLMs can still produce visually ungrounded or biased responses, motivating better alignment. The paper builds GPT-4V-annotated VLFeedback and distills its preferences into Qwen-VL-Chat; Silkie improves benchmark performance, including on MME and MMHal-Bench.

  • Problem

    Open LVLMs can generate misleading responses unanchored to visual context or biased outputs, despite advances in perceptual and cognitive abilities.

    Source: §1 · p. 1

  • Method

    The authors build VLFeedback from multimodal instructions and responses from 12 LVLMs, annotate outputs with GPT-4V, and use DPO to distill preferences into Qwen-VL-Chat.

    Source: §1 · p. 2

  • Results

    On MME, Silkie improves perception by 6.9% and cognition by 9.5%, and reaches a state-of-the-art 3.02 on MMHal-Bench.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VLFeedback particularly benefits fine-grained perception and complex cognition, with more consistent improvements than human-annotated preference data.

    Source: §1 · p. 2

    VLFeedback lacks sufficient supervision for safety alignment, despite including ethical considerations in its annotation design.

    Source: LIMITATIONS · p. 11

Abstract

from arXiv · show

This paper explores preference distillation for large vision language models (LVLMs), improving their ability to generate helpful and faithful responses anchoring the visual context. We first build a vision-language feedback (VLFeedback) dataset utilizing AI annotation. Specifically, responses are generated by models sampled from 12 LVLMs, conditioned on multi-modal instructions sourced from various datasets. We adopt GPT-4V to assess the generated outputs regarding helpfulness, visual faithfulness, and ethical considerations. Furthermore, the preference supervision is distilled into Qwen-VL-Chat through the direct preference optimization (DPO) method. The resulting model Silkie, achieves 6.9% and 9.5% relative improvement on the MME benchmark regarding the perception and cognition capabilities, respectively. Silkie also demonstrates reduced hallucination by setting a new state-of-the-art score of 3.02 on the MMHal-Bench benchmark. Further analysis shows that DPO with our VLFeedback dataset mainly boosts the fine-grained perception and complex cognition abilities of LVLMs, leading to more comprehensive improvements compared to human-annotated preference datasets.

1 INTRODUCTION

Open LVLMs can produce visually ungrounded or biased responses despite growing perceptual abilities. This work builds GPT-4V-annotated VLFeedback and applies preference distillation to improve LVLM alignment.

1 INTRODUCTION

  • Despite advances in perception and reasoning, open LVLMs can still produce misleading content unanchored to images or biased responses.The introduction presents addressing these misalignment issues as important to improving reliability and mitigating biased outputs.Source: §1 · p. 1
  • VLFeedback combines 80k multimodal instructions covering 65k unique images with responses sampled from a 12-LVLM pool, selecting four models per instruction.The dataset’s contribution is a large-scale multimodal preference resource annotated by GPT-4V.Source: §1 · pp. 2–3
  • GPT-4V assesses responses for helpfulness, visual faithfulness, and ethical considerations using the image, instruction, and evaluation templates.The resulting annotations include ratings and rationales.Source: §1 · p. 2, Figure 1 · p. 2
  • On MME, Silkie improves perception by 6.9% and cognition by 9.5%; it also sets a 3.02 state-of-the-art MMHal-Bench score.The paper also reports stronger gains on fine-grained perception and complex cognition tasks.Source: §1 · p. 2
  • DPO on VLFeedback distills preferences into Silkie, built on Qwen-VL-Chat, and improves LVLM capabilities across multimodal tasks.The analysis reports particularly pronounced gains in fine-grained perception and complex cognition, and more consistent boosts than human-annotated preference data.Source: §1 · pp. 2–3

2 VISUAL-LANGUAGE FEEDBACK DATASET

Section 2 constructs VLFeedback from diverse multimodal instructions, responses sampled from a 12-model LVLM pool, and GPT-4V ratings across three aspects. Helpfulness and visual-faithfulness scores have similar distributions, while ethical considerations arise for only a limited portion of sampled instructions.

2.1 INSTRUCTION SOURCE

  • Instruction sources cover general conversation and reasoning, academic vision-language tasks, robustness challenges, and domain-specific images and applications.The sources include misleading queries, text-rich and biomedical images, and embodied visual decision-making instructions.Source: §2.1 · p. 3

2.2 MODEL POOL

  • The model pool contains 12 LVLMs, with four models randomly sampled to decode each instruction.Decoding hyperparameters follow the original implementations' recommendations.Source: §2.2 · pp. 3–4

2.4 PREFERENCE STATISTICS

  • Helpfulness and visual-faithfulness scores have similar distributions, while only a limited portion of sampled instructions raise ethical considerations.The authors suggest the sampled instructions are not primarily red-teaming prompts and call for more targeted ethical-preference annotation.Source: Figure 2 · p. 5, §2.4 · p. 5

2.3 GPT-4V AIDED PREFERENCE ANNOTATION

  • GPT-4V rates helpfulness, ethical considerations, and visual faithfulness, which measures response consistency across modalities given the image and instruction.Visual faithfulness is rated on a 1-to-5 scale.Source: §2.3 · p. 5

2.4 PREFERENCE STATISTICS

  • GPT-4V rates higher than open-sourced LVLMs across the three aspects, though comparisons may be unfair for models trained on the annotated instructions and GPT-4V may favor its own characteristics.The authors note possible bias from GPT-4V's role as annotator and its proclivity for detailed responses.Source: §2.4 · p. 5, Table 3 · p. 6
  • Human annotators and GPT-4V achieved 83.1% average agreement on 100 randomly sampled response comparisons.Three human annotators compared response pairs using the same annotation guide as GPT-4V.Source: §2.4 · p. 6

3 PREFERENCE DISTILLATION FOR LVLMS

The section formulates preference distillation for LVLMs as learning from GPT-4V-annotated multimodal responses. It converts aspect scores into response preferences and uses DPO to fine-tune the target model directly on those preferences.

Motivation

  • The method explores whether preference data can improve LVLM helpfulness and visual faithfulness after instruction tuning has already equipped open-source models with promising abilities.Source: §3 · p. 6

Task Formulation

  • GPT-4V scores multimodal responses on three aspects; their averaged ratings rank responses, producing chosen–rejected pairs while tied scores are discarded.Each final preference example contains a prompt and two responses.Source: §3 · p. 6

Caveat

  • The authors report examples of GPT-4V annotations that are incorrect because of visual misjudgment or conflicting rationales.Source: §3 · p. 6

Preference Alignment Optimization

  • DPO fine-tunes the target model directly on preference data, bypassing the reward-modeling stage of the conventional RLHF pipeline.The paper describes conventional RLHF as learning a reward model and then optimizing with online reinforcement learning.Source: §3 · pp. 6–7

4 EXPERIMENTS

The experiments evaluate Silkie against benchmarks and baselines, then analyze where its improvements arise and how they compare with human-annotated preferences. Results show gains across benchmarks, with analysis highlighting fine-grained perception and complex cognition tasks.

4.1 EXPERIMENTAL SETTINGS

  • Experiments fine-tune Qwen-VL-Chat-v1.1 (7B) with LoRA for three epochs and evaluate perception, cognition, and visual faithfulness against open-source baselines and same-backbone variants.Source: §4.1 · pp. 7–8

4.2 RESULTS

  • MME perception rose from 1440.5 to 1539.6 and cognition from 362.5 to 397.1; MM-Vet improved 9.1%, while MMHal-Bench rose from 2.89 to 3.02.The reported comparison is Silkie versus the original Qwen-VL-Chat model.Source: Table 4 · p. 8, §4.2 · p. 8
  • Compared with heuristic baselines, Length as Best underperforms the original model, while GPT-4V as Best does not consistently match preference distillation; its MME cognition score decreases despite higher perception.Silkie also performs better than direct ShareGPT4V captioning-data training on MM-Vet and MME perception.Source: §4.2 · p. 8

4.3 ANALYSIS

  • Across MME tasks, DPO gains are strongest for OCR and artwork understanding in perception, and code reasoning and translation in cognition, rather than basic color or position recognition.The authors characterize these gains as emphasizing fine-grained perception and complex cognition.Source: §4.3 · p. 9, Figure 4 · p. 9
  • Compared with RLHF-V, the 1.4k-prompt VLFeedback subset improved MME perception and MM-Vet, while cognition and MMHal-Bench stayed consistent; RLHF-V hurt cognition and MM-Vet.The paper attributes the limited subset’s unchanged results to small coverage and describes RLHF-V’s scope as narrow.Source: §4.3 · p. 9, Figure 4 · p. 9
  • Silkie corrects Qwen-VL-Chat’s false denial of a flower vase on wooden stools, answers a science question correctly, and better structures a process-diagram report in Appendix B.Source: §4.3 · p. 9, Figure 5 · p. 10

5 RELATED WORKS

The related work places Silkie at the intersection of LVLM development and preference alignment. The paper uses GPT-4V to rank LVLM responses and distills those preferences into Qwen-VL-Chat.

Large Vision Language Models

  • LVLM approaches align language models with visual encoders through methods including Q-Former bridging, instruction integration, projection layers, and LoRA tuning.Source: §5 · p. 10

Preference Alignment

  • Preference alignment work spans supervised instruction tuning and feedback-based learning, with PPO and DPO as representative RLHF implementations.Prior LVLM work includes human-annotated preference datasets aimed at factuality and hallucination reduction.Source: §5 · p. 10
  • Unlike prior human-annotated LVLM preference work described here, this paper uses GPT-4V to rank responses and builds Silkie through preference distillation.Source: §5 · pp. 10–11

6 CONCLUSIONS

The paper presents VLFeedback and Silkie as resources for preference-distilled LVLM alignment, reporting improvements across benchmarks and particular gains in fine-grained perception and cognition.

6 CONCLUSIONS

  • 80k multi-modal instructions, responses from 12 LVLMs, and GPT-4V preference annotations make up VLFeedback.The paper describes the dataset as a resource for preference distillation.Source: §6 · p. 11
  • Silkie shows a substantial performance boost across various benchmarks compared with the original base model.Source: §6 · p. 11
  • Further analysis finds VLFeedback particularly advantageous for refining LVLMs’ fine-grained perception and cognition abilities.Source: §6 · p. 11
  • The paper reports greater scalability and more comprehensive improvements than human-annotated preference datasets.Source: §6 · p. 11

LIMITATIONS

The authors identify gaps in safety-oriented feedback and coverage of LVLMs and instruction datasets as limitations of the work.

LIMITATIONS

  • VLFeedback lacks sufficient supervision for safety alignment despite annotation addressing potential ethical considerations.The authors suggest adding safety-oriented instructions generated through red-teaming.Source: LIMITATIONS · p. 11
  • The work covers a limited range of LVLMs and instruction datasets amid a rapidly evolving landscape.The authors identify expanding coverage to newer models and more diverse datasets as future work.Source: LIMITATIONS · p. 11

A ANNOTATION TEMPLATES

The supplied material identifies two GPT-4V annotation-guideline tables and a case study of report composition.

A ANNOTATION TEMPLATES

  • Tables 5 and 6 are GPT-4V annotation guidelines for helpfulness and ethical considerations.Source: Table 5 · p. 15, Table 6 · p. 15
  • Figure 6 shows Silkie producing a comprehensive report that meets the word requirement with a more readable layout.Source: Figure 6 · p. 16

B CASE STUDY

In a challenging report-composition query, Silkie meets the word requirement and produces a more readable layout, while Qwen-VL-Chat does not meet the requirement.

B CASE STUDY

  • Silkie meets the word requirement and provides a more readable layout, whereas Qwen-VL-Chat’s relevant report falls short of the requirement.The query asks models to identify key processes and technologies in an image and compose a report.Source: B CASE STUDY · p. 16

C HUMAN EVALUATION

Human annotators and GPT-4V disagreed on both examples, with GPT-4V missing some subtle visual details. The examples also suggest inconsistent criteria for how visual faithfulness affects helpfulness scores.

C HUMAN EVALUATION

  • In the clock example, GPT-4V rejected human annotators’ view that the answer correctly described the clock and its location.Table 7 is an example of GPT-4V annotation for an LRV sample.Source: C HUMAN EVALUATION · p. 16, Table 7 · p. 17
  • In the second example, GPT-4V missed the ear-tag hallucination and language-quality concerns identified by all human annotators.Table 8 is an example of GPT-4V annotation for an SVIT sample.Source: C HUMAN EVALUATION · p. 16, Table 8 · p. 18
  • The examples indicate that GPT-4V may apply inconsistent criteria when visual faithfulness contributes to rejected answers’ helpfulness scores.Source: C HUMAN EVALUATION · p. 16
Loading 2312.10665v1…