What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning

1 Introduction

• Visual reasoning instructions improve performance more than image-captioning and visual-question-answering instructions.The empirical study examines instruction types and characteristics across six datasets and two MLLMs.Source: §1 · p. 2
• Increasing instruction complexity is more helpful than increasing diversity or adding fine-grained spatial information, and greater complexity correlates with higher average benchmark performance.The reported relationship follows an approximate linear trend.Source: Figure 1 · p. 1, §1 · p. 2
• ComVint uses a synthesize-complicate-reformulate pipeline to increase instruction complexity while checking accuracy and reformulating instructions into multiple formats.The complication stage uses image content and outside knowledge.Source: §1 · p. 2
• 27.86% and 27.60% improvements for LLaVA on MME-Perception and MME-Cognition, respectively, accompany ComVint's gains across four MLLMs on SEED-Bench and MME.The paper reports that ComVint outperforms existing visual instruction collections.Source: §1 · p. 2

2 Background

• Visual instruction tuning adapts instruction tuning to MLLMs, which generate output text conditioned on an image and a textual task instruction.The paper describes MLLMs as combining an image encoder, an LLM, and a connection module.Source: §2 · p. 2

2.1 Visual Instruction Collections

• Existing visual instruction collections address image captioning, visual question answering, and visual reasoning; the study separates LLaVA-Instruct into subsets for these task types.The subsets are LLaVA-Caption, LLaVA-VQA, and LLaVA-Reasoning.Source: §2.1 · p. 3
• Collections also vary in task diversity, instruction complexity, and fine-grained spatial information, which the paper treats as instruction characteristics.The reviewed collections include LRV, A-OKVQA, and Shikra-RD as examples with diverse task types, outside knowledge, and spatial information, respectively.Source: §2.1 · p. 3

3.1 Experiment Setup

• The analysis fine-tuned BLIP-2 and MiniGPT-4 on selected instruction collections, choosing models with minimal or no prior visual instruction tuning.BLIP-2 had not been tuned on visual instructions; MiniGPT-4 had been fine-tuned on 3,500 image captions.Source: §3 · p. 3, §3.1 · p. 3
• Evaluation used MME ACC+ and SEED-Bench Image ACC, restricting SEED-Bench evaluation to image-understanding tasks.MME ACC+ requires both questions in an instance to be answered correctly.Source: §3.1 · pp. 3–4

3.2 Results and Analysis

• Results were organized into task-type comparisons—captioning, VQA, and reasoning—and instruction-characteristic comparisons—diversity, complexity, and spatial annotation.Table 2 reports SEED-Bench and MME results for the two fine-tuned models across instruction collections.Source: §3.2 · p. 4, Table 2 · p. 4
• Visual reasoning outperformed image captioning and visual question answering; captioning degraded performance across benchmarks, while VQA was intermediate in most cases.The authors suggest the performance advantage may correlate positively with instruction difficulty.Source: §3.2 · p. 4, Table 2 · p. 4
• A-OKVQA achieved the highest accuracy across benchmarks, while LRV and Shikra-RD had minimal or negative impacts.The reported comparison emphasizes complexity over task diversity and fine-grained spatial information.Source: §3.2 · p. 4, Table 2 · p. 4
• Reasoning-oriented and complexity-enhanced instruction sets were particularly useful, but existing sets had limits that motivate automatic complex-instruction generation at scale.LLaVA-Rea had limited complexity, while A-OKVQA’s complexity was constrained by human annotators’ abilities.Source: §3.2 · p. 4

4 Approach

• The overall approach synthesizes primary reasoning instructions, iteratively complicates and verifies them, then merges and reformulates them into the final dataset.The three-stage pipeline is also depicted in Figure 2.Source: §4 · p. 4, Figure 2 · p. 5

4.1 Visual Reasoning Instruction Synthesis

• The synthesis stage creates cross-modal and outside-knowledge reasoning instructions to capture image content and external knowledge, respectively.These are the two visual reasoning task types targeted by the approach.Source: §4.1 · pp. 4–5

4.1.1 Cross-modal Reasoning Instructions.

• For cross-modal reasoning, informative Flickr30k Entities images supply captions and objects for GPT-4 to generate three diverse instructions.Images are filtered by caption length, retaining those whose five captions exceed 700 characters; prompt requirements and demonstrations aim to reduce simplistic or irrelevant instructions.Source: §4.1.1 · p. 5

4.1.2 Outside-knowledge Reasoning Instruction.

• For outside-knowledge reasoning, Visual Genome objects are ranked by IDF, and the highest-ranked object becomes the topic entity for instruction generation.Images with more than a set number of object annotations, such as seven, are removed; sampled OK-VQA categories guide balanced knowledge coverage.Source: §4.1.2 · pp. 5–6

4.2 Visual Reasoning Instruction Complication

• Because initial synthetic instructions can be simple or hallucinated, an iterative complicate-then-verify procedure raises complexity and filters contradictions with image annotations.The authors report that one or two complication turns are often sufficient; ChatGPT verification discards instructions that fail alignment checks.Source: §4.2 · p. 6

4.3 Visual Reasoning Instruction Reformulation

• Open-ended responses may not suit specific formats, so ChatGPT reformulates sampled instructions as boolean or multiple-choice QA and combines them with the originals.Human review found most instructions high quality, and ComVint examples were more complex and involved more reasoning steps than examples in LLaVA-Reasoning and LRV.Source: §4.3 · p. 6

5.1 Experimental Setup

• The four-model evaluation compares ComVint with six other instruction collections across SEED-Bench, MME, and three traditional VQA benchmarks.The models are BLIP-2, MiniGPT-4, LLaVA, and InstructBLIP.Source: §5.1 · pp. 6–7

5.2 Main Results

• Complex visual reasoning collections generally produce the largest improvements across three evaluation dimensions, though gains are smaller for models previously tuned on those data.InstructBLIP and LLaVA had already used A-OKVQA and LLaVA Reasoning during instruction tuning.Source: §5.2 · p. 7
• ComVint balances capabilities and improves performance across benchmarks, while LRV can outperform it on SEED-Bench but substantially reduce BLIP-2’s MME scores.BLIP-2’s MME-Perception falls from 1151.26 to 643.02 and MME-Cognition from 241.07 to 216.43 after fine-tuning on LRV.Source: §5.2 · p. 7
• ComVint is the only tested instruction collection that further boosts InstructBLIP on OK-VQA, A-OKVQA, and GQA; the other collections hurt performance.Source: §5.2 · pp. 7–8

5.3 In-Depth Analysis

• Human review found most sampled instructions accurate, with ComVint responses comparable to LLaVA-Reasoning and better than LRV.Three authors assessed 100 sampled instances; their correctness ratings had Fleiss’ Kappa of 0.91.Source: §5.3 · p. 8
• Across datasets, greater instruction complexity tracks roughly linearly with higher average benchmark performance, and ComVint is highest on both measures.Complexity was estimated from average reasoning-step counts.Source: §5.3 · p. 8
• Removing complication degrades LLaVA performance across all benchmarks, while removing either reasoning type hurts SEED-Bench and MME-Perception.On MME-Cognition, removing cross-modal reasoning yields the best result; using both reasoning types gives the best average performance.Source: §5.3 · p. 8, Table 6 · p. 8
• Removing reformulation reduces performance compared with diverse instruction formats, although open-ended instructions alone still significantly improve performance.Source: §5.3 · p. 8

• The conclusion identifies visual reasoning task type and instruction complexity as especially useful for improving MLLM capabilities.Source: §6 · p. 9

7 Limitation

• The study leaves interactions among training data, model architecture, and training strategies for future exploration.Source: §7 · p. 9
• The study focuses on training data while leaving model architecture and training strategies unchanged.It plans to explore how these factors coordinate with training data in future work.Source: §7 · p. 9
• The authors consider ComVint too small, despite reporting strong results with limited data.They believe scaling high-quality visual instruction data will further enhance performance.Source: §7 · p. 9

A Case Study

• ComVint’s sample asks for identifying three women by clothing color, their actions, and then their similarities and differences.The LLaVA-Reasoning example emphasizes scene description, while LRV focuses on object recognition.Source: A Case Study · p. 12
• Figure 4 contrasts sampled instructions from LLaVA-Reasoning, LRV, and ComVint, with ComVint’s instruction described as more complex.Source: Figure 4 · p. 12, A Case Study · p. 12

B Baseline Models

• BLIP-2 and InstructBLIP use FLAN-T5-XXL with only the Q-Former trainable, whereas MiniGPT-4 updates a connecting linear layer.LLaVA updates its linear layers and LLM while freezing the vision encoder.Source: B Baseline Models · p. 12

C Discussion

• ComVint, the most complex instruction set, achieves the best results, while LLaVA-Cap has the lowest complexity and accuracy.The results across four MLLMs and the MME and SEED-Bench benchmarks show an overall trend approximated by a linear function.Source: C Discussion · p. 12
• Higher instruction complexity overall corresponds to improved MLLM performance, though LRV is a possible exception.The authors suggest LRV’s inclusion of more incorrect instructions as a possible explanation.Source: C Discussion · p. 12
• LLaVA-Rea and A-OKVQA, with higher complexity, show better performance than LLaVA-Cap in the reported comparison.Source: C Discussion · p. 12

D More Experimental Results

• ComVint improves LLaVA performance on every SEED-Bench Image subtask, especially instance attributes, instance interaction, visual reasoning, and text recognition.Table 7 reports performance across all SEED-Bench Image subtasks.Source: D More Experimental Results · p. 12, Table 7 · p. 13

E Prompt Design

• The appendix identifies the prompts used to synthesize visual instructions and construct the later stages.Source: E Prompt Design · p. 13
• The prompt-design section provides prompts for instruction synthesis, complication, verification, and reformulation.The synthesis and reformulation prompts appear in Figures 5–6 and 9–10, complication in Figure 7, and verification in Figure 8.Source: E Prompt Design · p. 13

F Data Quality Evaluation Criteria

• Three authors manually assess visual-instruction quality, with correctness evaluated for both instructions and responses.The criteria are introduced as the manual assessment procedure in Section 5.3.Source: F Data Quality Evaluation Criteria · p. 13
• Instructions must require image information, be clear and interpretable, and align unambiguously with the image context.Source: F Data Quality Evaluation Criteria · p. 13
• Responses must avoid image contradictions and hallucinations, follow the instruction accurately, and remain factually and logically coherent.Source: F Data Quality Evaluation Criteria · p. 13
• Appendix figures provide prompts for cross-modal and outside-knowledge reasoning, instruction complication and verification, and Bool and multiple-choice QA generation.Source: Figure 5 · p. 14, Figure 6 · p. 15, Figure 7 · p. 16