PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models

Yupan Huang, Zaiqiao Meng, Fangyu Liu, Yixuan Su, Nigel Collier, Yutong Lu

arXiv:2308.16463v3cs.CVcs.CL

TL;DR

Multimodal instruction-following models struggle with coherence across multiple images, while specialized data for word-level multi-image dialogue is lacking. The paper introduces SparklesDialogue, SparklesEval, and SparklesChat, and reports improved multi-image understanding without compromising single-image capabilities.

  • Problem

    Existing multimodal instruction-following models struggle to capture interactions among multiple images and text, and specialized datasets for word-level multi-image dialogue are lacking.

    Source: §1 · p. 3

  • Method

    The paper creates SparklesDialogue, a GPT-assisted dataset for word-level interleaved multi-image dialogue, introduces the SparklesEval benchmark, and presents SparklesChat for open-ended multi-image conversations.

    Source: §1 · p. 3

  • Results

    SparklesChat improves multi-image understanding across BISON, NLVR2, and SparklesEval, while training with SparklesDialogue maintains and may enhance single-image capabilities.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The work concludes that multimodal instruction-following models can handle open-ended dialogues involving multiple images, with natural conversational flow and direct context awareness.

    Source: §7 · p. 9

    SparklesChat inherits LLM limits, including outdated knowledge, occasional inaccuracies, and limited context length and inference speed, as well as vision-model errors in recognition and visual-relation reasoning.

    Source: §7 · p. 9

Abstract

from arXiv · show

Large language models exhibit enhanced zero-shot performance on various tasks when fine-tuned with instruction-following data. Multimodal instruction-following models extend these capabilities by integrating both text and images. However, existing models such as MiniGPT-4 and LLaVA face challenges in maintaining dialogue coherence in scenarios involving multiple images. A primary reason is the lack of a specialized dataset for this critical application. To bridge these gaps, we introduce SparklesDialogue, the first machine-generated dialogue dataset tailored for word-level interleaved multi-image and text interactions. Furthermore, we construct SparklesEval, a GPT-assisted benchmark for quantitatively assessing a model's conversational competence across multiple images and dialogue turns. We then present SparklesChat, a multimodal instruction-following model for open-ended dialogues across multiple images. Our experiments validate the effectiveness of training SparklesChat with SparklesDialogue based on MiniGPT-4 and LLaVA-v1.5, which enhances comprehension across multiple images and dialogue turns, and does not compromise single-image understanding capabilities. Qualitative evaluations further demonstrate SparklesChat's generality in handling real-world applications. All resources related to this study are publicly available at https://github.com/HYPJUDY/Sparkles.

1 INTRODUCTION

Multimodal instruction-following models struggle to maintain coherence when users discuss multiple images, and specialized data for word-level multi-image dialogue is lacking. The paper introduces SparklesDialogue, SparklesEval, and SparklesChat to address this setting.

1 INTRODUCTION

  • Existing multimodal instruction-following models struggle to capture interactions among multiple images and text, mixing image content and losing coherence in open dialogue.This capability matters when users refer to multiple images with text snippets to convey detailed instructions.Source: §1 · p. 3
  • A key gap is the lack of specialized datasets for multimodal dialogue involving multiple images and fine-grained, word-level text interactions.The cited passage contrasts instruction-following needs with models trained to predict the next word on large web datasets.Source: §1 · p. 3
  • SparklesDialogue is introduced as the first machine-generated dataset for word-level interleaved multi-image and text interactions, generated with GPT-4 from detailed image descriptions.Its two subsets draw on different image and description sources to support robustness and diversity.Source: §1 · p. 3
  • SparklesEval is a GPT-assisted benchmark that scores open-ended, multi-image dialogue across image reasoning, cross-image and cross-turn coherence, and response relevance and completeness.These three criteria define the benchmark's conversational competence assessment.Source: §1 · p. 3
  • SparklesChat places image representations between text according to their positions in dialogue, unlike approaches that pair one image representation with sentence-level text.It is designed for open-ended dialogue across multiple images.Source: §1 · p. 3
  • SparklesChat improves multi-image accuracy on BISON and NLVR2 for MiniGPT-4 and LLaVA-v1.5, scores 8.56 out of 10 on SparklesEval, and maintains or potentially enhances single-image understanding.Qualitative evaluations also demonstrate applicability to real-world multi-turn dialogues involving multiple images.Source: §1 · p. 3

2 RELATED WORKS

Related work covers multimodal alignment data and models, image-dialogue datasets, and multimodal instruction tuning. The paper positions its work around complex interactions between multiple images and word-level text.

2 RELATED WORKS

  • Interleaved web datasets such as Common Crawl Interleaved data and Multimodal C4 combine multiple images and sentences, while Flamingo and Kosmos-1 adapt to tasks using image-text examples but may fall short on intricate instructions.The cited work attributes this limitation to next-word prediction training rather than training to perform the user's requested task (Ouyang et al., 2022).Source: §2 · pp. 3–4
  • Visual Dialog and LLaVA data focus on image-driven questions, while OpenViDial and PhotoChat sparsely integrate images into daily conversation; these datasets are not designed for in-depth instructive multi-image analysis.The passage identifies real-world analytical scenarios as a challenge for these dataset types.Source: §2 · p. 4
  • Multimodal instruction tuning uses diverse-task datasets and LLM-generated instruction data, and this work explores complex interactions between multiple images and word-level text.The cited examples include MultiInstruct, MiniGPT-4, LLaVA, and SVIT.Source: §2 · p. 4

3 SPARKLESDIALOGUE

SparklesDialogue is a machine-generated dataset built to improve multimodal conversation across images and turns. GPT-4 generates dialogues from image descriptions, using demonstrations and image-description pools, and the dataset combines two source-based subsets.

3 SPARKLESDIALOGUE

  • GPT-4 simulates realistic, diverse multi-turn conversations in which users introduce images and assistants give detailed responses with visual reasoning.Subsequent turns add images for further discussion; the data aims to cover real-world concepts and requests such as advice, guidance, and assistance.Source: §3 · pp. 4–5
  • Dialogue Demonstrations guide well-formatted, diverse responses, while Candidate Image Descriptions provide image-selection pools using descriptions sourced from annotations such as captions, boxes, and regions.The paper uses descriptions because GPT-4-Vision can have limitations in image understanding, including spatial reasoning and counting.Source: §3 · p. 5
  • Combining SparklesDialogueCC and SparklesDialogueVG improves models’ capacity for understanding and reasoning across images and text.The two subsets draw on different sources, balancing VG's human-annotated descriptions with CC's larger image pool and more hallucination-prone descriptions.Source: §3 · pp. 5–6
  • SparklesDialogueCC contains 4.5K dialogues with at least two images across two turns, while SparklesDialogueVG contains 2K dialogues with at least three distinct images across two turns.Figure 4 presents SparklesDialogueVG as a representative subset for dataset characteristics.Source: Figure 4 · p. 5, §3 · p. 6, Table 1 · p. 6
  • SparklesDialogueVG spans varied query types and real-world topics, and its high average assistant-message word count indicates thorough, detailed responses.The visualized characteristics include root verb–noun pairs and a word cloud of topics.Source: Figure 4 · p. 5, §3 · p. 6

4 SPARKLESEVAL

SparklesEval is a GPT-assisted benchmark for quantitatively assessing conversational competence in multi-image, multi-turn dialogues. It evaluates dialogue responses with three criteria and a combined score, using one dialogue per prompt.

4 SPARKLESEVAL

  • The benchmark addresses the subjectivity, cost, and time demands of human evaluation with GPT-assisted quantitative assessment.Its development was motivated by reported consistency between recent LLMs and human quality assessment.Source: §4 · p. 6
  • SparklesEval assesses Image Understanding and Reasoning, Cross-Image and Cross-Turn Coherence, and Relevance and Completeness of Responses.A judge model receives the complete dialogue and related image descriptions, then rates generated responses from 1 to 10 on each criterion.Source: §4 · p. 6
  • Averaging criterion ratings yields turn-specific scores and an overall score derived from those turn scores.The judge considers the current question and preceding dialogue history but is asked to assess only model-generated responses.Source: §4 · p. 6
  • Assessing one dialogue per prompt avoids the position bias associated with comparing multiple dialogues in a prompt and avoids recalculating combined scores for multiple dialogues.The approach uses a combined score rather than a singular score for a more comprehensive and interpretable assessment.Source: §4 · p. 6
  • SparklesEval contains 150 dialogues, one-third of which include two images in the first and second conversational turns.It uses detailed, human-annotated SVIT image descriptions and questions selected for unique verb-noun combinations.Source: §4 · p. 7

5 SPARKLESCHAT

SparklesChat adapts multimodal instruction-following architectures to dialogue inputs that interleave images and text. Its training samples represent turns as question-answer pairs and train the model to predict responses from the dialogue context.

5 SPARKLESCHAT

  • SparklesChat places different image representations between text according to their positions in the dialogue, rather than supplying one image followed by a sentence embedding.It can use MiniGPT-4 or LLaVA, which connect a pretrained vision encoder and language decoder with a projection layer.Source: §5 · p. 7
  • For instruction tuning, each turn of a T-turn dialogue is simplified into a question-answer pair, and the pairs are sequenced with a predefined system prompt.The prompt for a turn incorporates the system prompt and dialogue content up to that turn.Source: §5 · p. 7
  • Training uses an autoregressive objective to predict target responses from prompts containing the dialogue context up to that turn.The prompt combines the system prompt with the dialogue content available at that turn.Source: §5 · p. 7

6 EXPERIMENTS

Experiments report gains in multi-image understanding for SparklesChat based on MiniGPT-4 and LLaVA-v1.5, while single-image results are not compromised overall. Ablations examine dialogue-turn sampling and dataset subsets, and demonstrations cover open-ended scenarios with multiple images and turns.

6.1 COMPARISON OF MODEL PERFORMANCE

  • The evaluation compares multi-image dialogue competence on SparklesEval with zero-shot image selection on BISON and visual reasoning on NLVR2.Table 3 compares model performance on multi-image and single-image understanding benchmarks.Source: §6.1 · pp. 7–8
  • SparklesChat based on MiniGPT-4 improves accuracies by 10.7% on BISON and 6.7% on NLVR2, and scores 8.56 out of 10 on SparklesEval.The evaluations cover binary image selection, visual reasoning, and conversational competence across multiple images.Source: §6.1 · p. 8
  • Fine-tuning LLaVA-v1.5 with SparklesDialogue using LoRA improves open-ended dialogue tasks and enhances BISON and NLVR2 results.The authors report these results as evidence of adaptability with minimal additional training cost.Source: §6.1 · p. 8
  • SparklesDialogue training consistently improves MiniGPT-4 across single-image benchmarks, while LLaVA-v1.5 has marginal reductions in some scores.The authors conclude that multi-image comprehension gains do not compromise single-image understanding overall and may augment it.Source: §6.1 · p. 8

6.2 ABLATION STUDIES

  • A 2:1 first-turn-to-second-turn sampling ratio is adopted for balanced performance across benchmarks.A 1:1 ratio is strongest on BISON and SparklesEval but decreases NLVR2 performance; increasing the second-turn ratio to 1:2 causes a performance drop.Source: §6.2 · pp. 8–9
  • SparklesDialogueVG outperforms SparklesDialogueCC on BISON and SparklesEval, performs similarly on NLVR2, and combining the subsets yields higher or comparable performance.The authors associate VG with higher-quality human-annotated data and CC with diversity, while noting that VG shares image and caption sources with SparklesEval.Source: §6.2 · p. 9

6.3 DEMONSTRATIONS AND APPLICATIONS

  • Demonstrations cover two- or three-turn free-form scenarios, with one to five images in each turn.The examples include creative requests such as stories, songs, dialogues, and titles.Source: §6.3 · p. 9

7 CONCLUSION AND LIMITATIONS

The paper concludes that SparklesDialogue, SparklesEval, and SparklesChat address open-ended multimodal dialogue across multiple images. It also describes turn-based instruction tuning and situates the work among multimodal datasets and instruction-following methods.

7 CONCLUSION AND LIMITATIONS

  • Experiments with SparklesDialogue-trained SparklesChat showed effectiveness on multi-image and single-image benchmarks, while qualitative demonstrations covered free-form applications.The conclusion reports results using both MiniGPT-4 and LLaVA-v1.5 architectures.Source: §7 · p. 9

A INSTRUCTION-TUNING DETAILS

  • Training samples sequence each dialogue turn’s user question and assistant answer under a system prompt, with the LLM trained autoregressively on response tokens conditioned on preceding-turn prompt and response tokens.For each dialogue, the paper constructs T training samples.Source: A INSTRUCTION-TUNING DETAILS · p. 14

B RELATED WORKS

  • The related-work review covers image-text alignment and interleaved web datasets, whose models can use multiple image-text examples but may fall short on intricate instructions.The paper positions SparklesDialogue as exploring interactions between multiple images and word-level text, with SparklesChat trained on that dataset.Source: B RELATED WORKS · pp. 15–16
  • Prior dialogue datasets include image-centered conversations and everyday conversations with sparse images, but the paper says these do not meet user-assistant instructive problem-solving needs.LLM-generated image-grounded datasets often elicit brief responses that may not fit in-depth multi-image analysis.Source: B RELATED WORKS · p. 15

C IMPLEMENTATION DETAILS

The implementation adapts SparklesChat to MiniGPT-4 and LLaVA-v1.5, with distinct training setups and a turn-based prompt-response format. The training objective predicts assistant answers rather than user-provided prompts.

C IMPLEMENTATION DETAILS

  • MiniGPT-4-based SparklesChat trains only the projection layer for 1,500 steps at batch size 8, using equal sampling from SparklesDialogueCC and SparklesDialogueVG and a 2:1 ratio of first- to second-turn data.The visual and language components other than the projection layer are frozen.Source: C IMPLEMENTATION DETAILS · p. 16
  • The LLaVA-v1.5 implementation adapts the codebase for multiple images and uses LoRA for 2,000 steps across four GPUs, with learning rate 2e-5 and batch size 4 per GPU.Its language model uses the 7-billion-parameter Vicuna architecture.Source: C IMPLEMENTATION DETAILS · p. 16
  • Evaluation uses temperature 1.0, top_p 0.9, max_new_tokens 300, and beam size 2 for demonstrations or 1 for other evaluations.The reported GPT-4 data-generation settings yield an approximate dataset construction cost of $500.Source: C IMPLEMENTATION DETAILS · p. 16
  • Table 5’s two-turn training format uses image tags, system and speaker messages, and separators; only assistant-answer sequences contribute to autoregressive loss, not the user-provided prompt.The image tags are replaced by visual features from a linear projection layer.Source: C IMPLEMENTATION DETAILS · p. 16, Table 5 · p. 17

D DETAILED MODEL COMPARISON ON MULTI-IMAGE EVALUATION

Across the reported SparklesEval criteria, reasoning-data training outperforms description-data training, and SparklesChat approaches GPT-4’s score. The comparison also reports BISON and NLVR2 evaluations.

D DETAILED MODEL COMPARISON ON MULTI-IMAGE EVALUATION

  • SparklesChat’s score is about 92% of GPT-4’s 9.26 on SparklesEval, with GPT-4’s result attributed largely to detailed ground-truth annotations.The paper reports that SparklesChat trained on reasoning data scores 6.71, compared with about 3 for description data.Source: D DETAILED MODEL COMPARISON ON… · p. 17
  • Reasoning-data-trained SparklesChat outperforms description-data-trained models on all SparklesEval metrics, with its highest scores in both dialogue turns.The reported criteria cover image understanding and reasoning, cross-image and cross-turn coherence, and response relevance and completeness.Source: D DETAILED MODEL COMPARISON ON… · p. 17
  • Training SparklesChat on reasoning data adapted from LLaVA improves performance over training on description data across all metrics, with its highest scores in both turns across all criteria.Source: D DETAILED MODEL COMPARISON ON… · p. 17

E DETAILED RESULTS OF ABLATION STUDIES

The ablation results are organized around dialogue-turn ratios and dataset subsets, while SparklesEval aggregates GPT-assisted ratings over three criteria and two turns.

E DETAILED RESULTS OF ABLATION STUDIES

  • Table 7 reports ablations varying SparklesDialogue’s dialogue-turn ratios and subsets, with BISON and NLVR2 accuracy and SparklesEval ratings as evaluation measures.The supplied passage identifies the ablation dimensions and table scope but provides no cell values.Source: E DETAILED RESULTS OF ABLATION STUDIES · p. 17, Table 7 · p. 18

F SPARKLESEVAL DETAILS

  • SparklesEval scores image understanding and reasoning, cross-image and cross-turn coherence, and response relevance and completeness.GPT models assign combined scores for each turn based on these criteria.Source: F SPARKLESEVAL DETAILS · p. 18
  • The overall SparklesEval score averages the two turn scores, each computed as the mean of the three criteria across evaluation samples.The authors do not use the judge models’ supplied combined scores because their calculations may be inaccurate.Source: F SPARKLESEVAL DETAILS · p. 18

G ZERO-SHOT EVALUATION ON VISION-LANGUAGE TASKS

The section evaluates zero-shot multi-image understanding through binary image selection and visual reasoning, using accuracy on BISON and NLVR2 without additional training.

G ZERO-SHOT EVALUATION ON VISION-LANGUAGE TASKS

  • BISON tests whether a model can select the image matching a text query, while NLVR2 tests whether a sentence is true of a pair of images.Both tasks use accuracy; the evaluation samples 150 examples for each dataset.Source: G ZERO-SHOT EVALUATION ON… · pp. 18, 20
  • The evaluation uses 150 randomly sampled COCO-BISON examples and 150 NLVR2 balanced-test examples, with overlapping training and evaluation images removed for SparklesDialogueVG.The BISON sample comes from COCO-BISON; the NLVR2 sample comes from the public balanced test set.Source: G ZERO-SHOT EVALUATION ON… · pp. 18, 20
  • Models receive no additional training and are prompted to reason step by step before giving an answer; malformed outputs are regenerated.The prompt combines reasoning and answer extraction in one stage, using “Therefore, the answer is” to identify the answer.Source: G ZERO-SHOT EVALUATION ON… · pp. 20–21

H DEMONSTRATIONS AND APPLICATIONS

Demonstrations show SparklesChat handling dialogue and creative requests involving multiple images, while evaluation reports strong judged performance and a limitation on complex concatenated queries.

H DEMONSTRATIONS AND APPLICATIONS

  • Examples demonstrate two- or three-turn dialogues with one or two images per turn, including stories, songs, and dialogues connecting places and characters.The demonstrations also include describing and reasoning about different image groups.Source: H DEMONSTRATIONS AND APPLICATIONS · p. 21, Figure 6 · p. 22, Figure 7 · p. 23
  • With four- or five-image queries, SparklesChat distinguishes images and follows complex instructions, but may ignore the last of three concatenated questions.The authors suggest that similar multi-question patterns are absent from training data and propose multiple turns per training dialogue as a possible remedy.Source: H DEMONSTRATIONS AND APPLICATIONS · p. 21

I JUDGING WITH DIFFERENT GPT MODELS

✨SparklesChat

  • 8.56/10: SparklesChat scores about 92% of GPT-4’s 9.26/10 score under the default judge, versus 3.91 for MiniGPT-4.The authors note that GPT-4’s score mainly reflects its use of detailed ground-truth annotations, and that LLM judges may show self-enhancement bias.Source: I JUDGING WITH DIFFERENT GPT MODELS · p. 25, Table 10 · p. 26

J CHARACTERISTICS AND VERB-NOUN DISTRIBUTION ANALYSIS

This section analyzes SparklesDialogue’s verb–noun distributions and describes GPT-assisted generation of SparklesDialogueVG and SparklesDialogueCC, with setups varying dialogue structure, demonstrations, and candidate-image selection. It also illustrates SparklesDialogueCC with generated conversations connecting visual details across images and turns.

J CHARACTERISTICS AND VERB-NOUN DISTRIBUTION ANALYSIS

  • Verb–noun analysis parses user messages, prioritizes the last sentence (falling back to the first), and plots the 20 most common root verbs with their four most common direct noun objects.The analysis uses the Berkeley Neural Parser and visualizes distributions by number of images per turn for SparklesDialogueVG and SparklesDialogueCC.Source: J CHARACTERISTICS AND VERB-NOUN… · pp. 26–27, Figure 12 · p. 28, Figure 13 · p. 29

K.1 SINGLE DIALOGUE GENERATION FOR SPARKLESDIALOGUEVG

  • SparklesDialogueVG generation creates two-turn dialogues with two or three first-turn images, using unique verb–noun demonstration questions and four randomly selected, non-reused candidates from roughly 100,000 image-text pairs.The demonstration pools contain 661 dialogues for two-image turns and 441 for three-image turns.Source: §K.1 · p. 27

K.2 MULTIPLE DIALOGUES GENERATION FOR SPARKLESDIALOGUECC

  • SparklesDialogueCC generation prompts GPT-4 for three dialogues per response, with one, two, or three first-turn images and one second-turn image, using three demonstrations and nine candidate image descriptions.The demonstrations were curated from 150 dialogues, evenly divided among the three first-turn image counts; the image-text dataset contains about 3,500 pairs.Source: §K.2 · pp. 35–36
  • The candidate image descriptions include interiors, a landmark, a logo, and a person, and the corresponding images were not sent to GPT-4 for data generation.Figure 18 identifies the images associated with the dialogue-generation examples.Source: §K.2 · pp. 39–41, Figure 18 · p. 40
  • A logo interpretation is extended into a discussion of whether the design complements the Eiffel Tower setting.The exchange considers the logo’s possible symbolism and how its modern style and red color might fit the landmark’s historic and lively context.Source: §K.2 · pp. 41–42
  • A multi-turn story links a dog, its pilot owner, an airplane, and a golf course into a continuing narrative.The golf-course scene is used to extend the story as a place where the pilot and dog reconnect.Source: §K.2 · p. 43
  • Another exchange combines kitchen, storage-room, and garden images to imagine a man’s temporary workspace and after-work routine.The temporary office is framed as an adaptation during renovation or a family visit, while the garden provides a place to unwind.Source: §K.2 · pp. 43–44

L DIALOGUE DEMONSTRATIONS’ CONTRIBUTION TO DATA QUALITY AND DIVERSITY

Dialogue demonstrations improve generated dialogue formatting and diversity by providing formatting guidance and conditioning different examples. Removing them produced responses that missed desired structural criteria and lacked typical diversity.

L DIALOGUE DEMONSTRATIONS’ CONTRIBUTION TO DATA QUALITY AND DIVERSITY

  • Without dialogue demonstrations, generated responses failed to assign image IDs, specify images per turn, or introduce new images in later turns.The responses also lacked diversity, and the authors conclude demonstrations are crucial for data quality and diversity.Source: L DIALOGUE DEMONSTRATIONS’… · p. 44
  • A kitchen example moves from detailed image description to follow-up suggestions about lighting and storage.The user first requests a description, then asks what could be improved; the assistant offers possible updates while noting that preferences and functionality needs matter.Source: L DIALOGUE DEMONSTRATIONS’… · pp. 44–45
  • A cross-image example asks how to combine a dog image with a plane-at-sunset scene, prompting suggestions for a connected story illustration.The response proposes placing the dog in the foreground and linking the scenes through elements such as a toy plane or a family member near the plane.Source: L DIALOGUE DEMONSTRATIONS’… · p. 45

Related papers

Loading 2308.16463v3…