Source-linked AI summary
Instruction Mining From Images: Constructing a Synthetic Dataset for Multimodal Learning ILSP DCVLR Competition Submission
Eleni E. Kasoura, Dimitrios Damianos, Georgios Paraskevopoulos, Vassilios Katsouros
TL;DR
The paper addresses the need for compact, clean, reproducible image-grounded instruction data for visual reasoning. It adapts Magpie-style synthesis into a pipeline with system prompts, sanitization, and reward-based filtering; the authors emphasize that sanitization and prompts are important because VLMs tend to produce captions without guidance.
Problem
The work asks whether a compact, high-quality, traceable image–instruction corpus can trigger visual reasoning without sacrificing efficiency, given concerns that existing resources prioritize scale over coverage, cleanliness, and reproducibility.
Source: §1 · p. 1Method
The pipeline adapts Magpie to generate image-conditioned instructions and responses, then sanitizes, tags, and filters them using reward scores.
Source: §1 · pp. 1–2Results
The authors report that proper sanitization and system-prompt conditioning are important because VLMs tend to produce captions and image descriptions without further guidance.
Source: §6 · pp. 4–5Takeaways & Limitations
The paper's conclusion is that visual instruction synthesis depends on prompt guidance and sanitization, with reward-model scores used to determine sample quality.
Source: §6 · pp. 4–5The work remains at proof-of-concept stage, with no systematic ablations and a 7-billion-parameter synthesis model that limits instruction quality and diversity.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
This work presents a compact and reproducible pipeline for synthesizing and curating image-based instruction-response pairs for DCVLR. We draw from the Cauldron collection and use Magpie-style instruction and response synthesis with Qwen2.5-VL-7B Instruct to create concise, image-conditioned instruction pairs. The pipeline consists of instruction synthesis, conditioned on input images and controlled via a system prompt, sanitization, tagging, and filtering of the instructions based on the score obtained by a reward model (DeBERTa-v3). We retain samples with positive scores to maximize the quality of the generated instructions. We observe that score distributions have a median and mean reward score of ~ 1.53 and ~ 1.75 respectively, indicating that most generated instructions are of good quality. The resulting dataset contains diverse instructions in the math, spatial, and social reasoning domains.
1 Introduction
The paper proposes a compact, traceable pipeline for image-grounded instruction–response synthesis, adapting Magpie to vision–language models. It combines image-conditioned generation with multi-image formatting and reward-based quality control.
1 Introduction
- The work targets a compact, high-quality, traceable image–instruction corpus that triggers visual reasoning without sacrificing efficiency.Source: §1 · p. 1
- The pipeline adapts Magpie to vision–language models, using Cauldron images and Qwen2.5-VL-7B-Instruct to generate instructions and paired answers.Generated instructions are tagged and scored with a reward model to retain high-scoring examples.Source: §1 · pp. 1–2
- A system prompt and image-only pre-query have the model synthesize the user instruction from the image before generating its paired assistant response.This produces grounded instruction–response pairs.Source: §1 · p. 2
- A Magpie-inspired vision-token format conditions on multiple images while preserving grounding and keeping interactions concise.Source: §1 · p. 2
- The reward-model stage uses a tunable threshold: stricter thresholds increase rejection rates while improving final instruction quality.Source: §1 · p. 2
2 Related Work
Related work motivates template-driven self-synthesis and reward-based filtering, then positions the paper's adaptation of these approaches for vision–language instruction data.
2 Related Work
- Magpie formalizes self-synthesis from chat-template context, while this work adapts it to VLMs with strict formatting and sanitization.Earlier approaches including Self-Instruct Wang et al. [2023], Alpaca Dubois et al. [2023], and Evol-Instruct/WizardLM Xu et al. [2024] are described as more limited in scalability and diversity.Source: §2 · p. 2
- Visual-instruction systems such as LLaVA Liu et al. [2023], InstructBLIP Dai et al. [2023], and MiniGPT-4 Zhu et al. are cited as showing VLM visual-reasoning and multitask capabilities.The passage highlights multi-image conditioning, text-rich inputs, and instruction diversity.Source: §2 · p. 2
- Reward models are used in alignment to filter or weigh data; this work scores instructions with DeBERTa-v3 He et al. and applies a reproducible threshold keep/cut rule.The passage characterizes the model as providing good performance and low latency.Source: §2 · p. 2
3 Methodology
The method turns images into concise, grounded instruction–response pairs through Magpie-style self-synthesis, sanitization, and reward-based selection.
3 Methodology
- An image-only pre-query prompts autoregressive instruction synthesis, followed by assistant-response generation after inserting the assistant header.The model is conditioned by the image and a short system prompt specifying a task family, such as spatial arrangement.Source: §3 · p. 2
- Sanitization removes boilerplate and residual formatting, standardizes punctuation, and accompanies visible-content grounding guards and near-duplicate removal.Instructions are then tagged, including with difficulty labels.Source: §3 · pp. 2–3
- A reward model assigns each pair a scalar score, and thresholding retains higher-scoring items while discarding lower-quality pairs.An optional reasoning check and final sanitization may precede acceptance; outputs include cleaned instructions, responses, tags, and scores for downstream use.Source: §3 · p. 3
4 Implementation details
The implementation samples Cauldron images and uses a reward model to select a final set. System prompts target several task categories and a general category.
4 Implementation details
- The implementation samples 5,000 images from each Cauldron subset and selects the top 10,000 samples with positive reward scores using threshold t = 0.It uses Qwen2.5-VL-7B-Instruct and OpenAssistant/reward-model-deberta-v3-large-v22; experiments ran on Leonardo-Booster nodes with four NVIDIA A100 GPUs each.Source: §4 · p. 3
4.1 System Prompts
- System prompts target text extraction, spatial relations, mathematics/quantitative reasoning, and a general category for most subsets.Source: §4.1 · p. 3
- Figure 2 presents the reward-score distribution of the selected samples.Source: Figure 2 · p. 4
- Table 1 reports subset counts for the generated dataset, which contains 10,000 samples in total.Source: Table 1 · p. 4
5 Dataset Statistics
Selected samples have mean and median reward scores of 1.7518 and 1.5341, respectively, and the final dataset is skewed toward visual question answering.
5 Dataset Statistics
- 1.7518 mean and 1.5341 median reward scores characterize the selected samples.Their scores range from approximately 0.5 to 6.5.Source: §5 · p. 4
- Selected-sample reward scores range from approximately 0.5 to 6.5.Source: §5 · p. 4
- The final dataset is skewed toward visual question answering; the authors suggest complex math, diagram, and infographic images may yield lower-quality instructions that the reward model rejects.Source: §5 · p. 4
6 Conclusions
The work extends Magpie to synthesize image-grounded visual instructions, with system prompts and reward scores guiding sample quality. Future work targets greater diversity and systematic ablations.
6 Conclusions
- The paper extends Magpie to generate visual instructions grounded in input images.Source: §6 · p. 4
- System prompts guide instruction synthesis, while reward-model scores automatically determine sample quality.Source: §6 · pp. 4–5
- Planned work includes broader system prompts and input images for greater diversity, plus detailed ablations of reward model, threshold, and prompt choices.Source: §6 · p. 5
Limitations
The authors describe the work as a proof of concept with limitations in evaluation and synthesis capacity, and identify possible steps toward greater dataset diversity.
Limitations
- The work is at the proof-of-concept stage, and systematic ablation studies were not performed because of time constraints.Source: Limitations · p. 5
- The relatively small 7-billion-parameter synthesis model limits the quality and diversity of generated visual instructions.Source: Limitations · p. 5
- Reward-score stratification and additional system prompts may be used to increase diversity in the final dataset.Source: Limitations · p. 5
Broader Impact
The authors describe synthetic data generation as offering research and privacy-related benefits while also posing misuse and bias risks. They call for safeguards and further study of diversity loss from synthetic-data training loops.
Broader Impact
- Scalable synthetic-data pipelines may accelerate machine-learning research, especially where data are relatively scarce.Source: Broader Impact · p. 5
- Synthetic data may support more private and fair models by excluding personal identifying information and allowing biases to be balanced.Source: Broader Impact · p. 5
- Risks include sophisticated disinformation and deepfakes, bias amplification, and reduced output diversity in self-consuming training loops; the authors call for detection, fairness audits, and ethical guidelines.Source: Broader Impact · p. 5
A System prompt categories
Table 2 presents two example prompts for each prompt category.
A System prompt categories
- Table 2 lists prompt categories with two example prompts per type.Source: Table 2 · p. 7