UniData: Universal Multimodal Instruction Generation Pipeline

1 Introduction

• Existing data generation relies heavily on costly manual collection and annotation, while many automated approaches target a specific scenario and do not generalize broadly.The introduction also describes prior universal approaches as limited in modality support and multi-round generation.Source: §1 · p. 1
• The pipeline expands user requirements into diverse events, then uses an any-to-any model to generate multi-round multimodal instructions iteratively.It also corrects instruction flow using correlations between previous and current rounds to minimize redundancy and irrelevance.Source: §1 · pp. 1–2
• UniDataset contains 20,000 instructions across nine modalities on both input and output sides, providing data for training UniData and other any-to-any models.The modalities include language, image, music, emoji, code, map, link, math, and QR code.Source: §1 · p. 2
• UniData’s generated data surpasses other pipelines on reasonableness, clarity, detail, and relevance, and can improve other multimodal models’ understanding and generation.The paper reports these outcomes in its experiments.Source: §1 · p. 2
• UniData generates multi-round multimodal instructions from one simple requirement.The pipeline is presented as a universal instruction-generation approach.Source: §1 · pp. 1–2

2 Related Work

• Earlier multimodal models largely produced language-only output, while SEED-X and Show-o explored image understanding and generation and NExT-GPT integrated more modalities.NExT-GPT’s listed modalities include images, audio, and video.Source: §2 · p. 3
• Training these models faces substantial data-availability challenges: NExT-GPT cannot provide training data due to copyright constraints, while Show-o is limited to training in an end-to-end way because no appropriate dataset is available.Source: §2 · p. 3
• Instruction-generation methods progressed from language-only Self-Instruct to visual VIGC and abstract-image generation in Multimodal Self-Instruct, but retained limited multimodal capabilities.The cited methods are Self-Instruct (Wang et al., 2023), VIGC (Wang et al., 2024a), and Multimodal Self-Instruct (Zhang et al., 2024).Source: §2 · p. 3
• UniData is designed to understand and generate nine types of multimodal information, broadening the stated scope of instruction generation.The paper frames this as a broader applicability than previous research.Source: §2 · p. 3

3 Dataset Construction

• Existing datasets are often single-modality, especially in their outputs; multimodal input-output datasets are described as limited in modality range, instruction rounds, size, or availability.UniDataset is presented as supporting inputs and outputs across nine modalities.Source: §3 · p. 3

3.1 Procedure of Data Construction

• Randomly paired topics and varied rounds, emotions, and occasions guide generation of 20,000 multi-round instructions.The selected round counts range from 5 to 30.Source: §3.1 · p. 3
• The procedure forecasts indicators for eight modalities beyond language and generates prompts or queries for corresponding multimodal content.The listed modalities include image, music, emoji, code, map, link, math, and QR code.Source: §3.1 · p. 4
• A toolkit using GPT-4o, Google Search, Google Maps, DALL·E 3, MusicGen, and the Apple Emoji Archive produces multimodal content from generated prompts, content, or queries.The figure summarizes this as the final content-generation stage.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 Data Format and Statistics

• Each sample maps a contextual triple to an instruction with nine typed modality slots, filled by dedicated generators using modality-specific cues.The language slot carries the dialogue text, while cues include prompts, labels, queries, or content.Source: §3.2 · p. 4

4.1 Expanding Diverse Events

• DEG expands limited keyword requirements into diverse events, with stochastic factors such as random emotions, occasions, and seeds broadening the event range.Each event corresponds to a distinct instruction.Source: §4.1 · p. 5

4.2 Generating Multimodal Instructions

• MIG combines a text-centred generator with modality-specific realization tools to understand multimodal content and iteratively generate each next-round instruction from one event and prior rounds.Previous multimodal instructions are converted into token embeddings in a unified feature space.Source: §4.2 · p. 5

4.3 Correcting Instruction Flow

• During long-term inference, the correction chain compares new instructions with previous rounds and raises the repetition penalty when a new instruction is too close to earlier ones.This addresses repetitive generation and hallucinations that can arise during inference.Source: §4.3 · p. 6

2. Irrelevant Generation: If the new instruction

• When an instruction is unrelated to previous ones, the pipeline raises temperature to introduce randomness and sample more alternatives.Source: §2 · p. 7
• If an instruction is neither repetitive nor irrelevant, the pipeline treats it as high-quality and proceeds to the next round.Source: §2 · p. 7
• R < 0.05 indicates irrelevant generation, R > 0.4 indicates repetitive generation, and values between these thresholds indicate high-quality generation.The authors set α1 = 0.05 and α2 = 0.4 empirically; users can adjust them dynamically.Source: §2 · p. 7

5 Experiments

• Evaluation uses GPT-Guided reasonableness, clarity, detail, and relevance metrics and tests whether generated instructions improve other multimodal models after fine-tuning.The study compares Self-Instruct and VIGC as instruction-generation baselines.Source: §5.1 · p. 7, §5.2 · p. 7

5.1 Instruction Quality Evaluation

• UniData surpasses both baselines on all four data-quality dimensions, with its largest margins on clarity and relevance.Table 1 reports quantitative data-quality results.Source: §5.1 · p. 7, Table 1 · p. 8

5.2 Post Fine-Tuning Assessment

• Generated instructions improve Ovis2-1B on MMMU, ChartQA, and TextVQA visual-understanding benchmarks.The study also evaluates image understanding and generation separately.Source: §5.2 · p. 8, Table 6 · p. 8
• 19.42% → 35.25% is the reported gain in the interleaved image setting; gains are smaller for single- and multi-image settings.The data also improves Image-to-Text and Text-to-Image tasks and helps generate relevant images from text.Source: §5.2 · p. 8, Table 5 · p. 8
• Music understanding and generation improve with small margins; all five understanding metrics move favorably, with clearest gains on METEOR and ROUGE.For music generation, FAD and KL decrease while CLAP increases.Source: §5.2 · p. 8, Table 2 · p. 8, Table 3 · p. 8
• 20.3 to 21.9 is the MMLU-Pro accuracy increase; the instructions also improve LLaMA 2 code generation and Ovis2-1B performance on both math benchmarks.The math gains are consistent in direction but modest in magnitude.Source: §5.2 · p. 8, Table 4 · p. 8

5.3 Ablation Study

• Without event expansion, instructions lose enriched prompt information, quality declines, and the number of generated instructions is substantially reduced.Three random generations show diverse event distributions, and examples from the same keywords yield distinct events.Source: §5.3 · p. 9, Figure 8 · p. 9

Effectiveness of Correcting Instruction Flow

• Flow correction improves data quality on all four dimensions and can avoid repetitive emoji and irrelevant code or “null” content.The comparison is shown in Table 7, with an example in Figure 9.Source: Effectiveness of Correcting… · p. 9, Figure 9 · p. 9

• UniData establishes a framework for scalable data acquisition for MLLMs.Source: §6 · p. 9
• The pipeline delivers high-quality, diverse, and cost-effective instruction sets.Source: §6 · p. 9
• The paper concludes that these instruction sets accelerate progress in multimodal learning paradigms and downstream applications.Source: §6 · p. 9

Limitations

• Per-modality quality is bounded by external realization tools, and long-horizon structured reasoning relies only on simple correction rules.Source: Limitations · p. 9
• The nine-modality coverage omits video, 3D, motion, tables, biological data, and spoken input.Source: Limitations · p. 9
• Biases from LLM backbones, search APIs, and modality generators may enter UniDataset and be amplified downstream; systematic bias audits remain future work.Source: Limitations · p. 9

A.1 Topic Pool

• The 90-domain topic pool spans everyday life, science and engineering, business, arts, and humanities; intersecting two sampled topics yields interdisciplinary prompts.The pool’s breadth is described as the primary driver of content diversity.Source: §A.1 · p. 13

A.2 Multimodal Toolkit

• The multimodal toolkit assigns off-the-shelf tools to individual modalities and favors public APIs and open-source models for reproducibility and extension.Table 9 summarizes each component, target modality, source, and role.Source: §A.2 · p. 13

A.3 Data Diversity

• 17.5 rounds per dialogue on average versus at most 4.8 for prior work; UniDataset supports nine modalities on both input and output, the largest set in this comparison.Source: §A.3 · p. 13

A.4 Human Oversight for Bias Mitigation

• Human oversight targets bias through balanced hand-selected seed topics and filtering that removes implausible or near-duplicate events.The event filtering is described as mitigating propagation of LLM-specific stylistic biases into final instructions.Source: §A.4 · p. 13

A.5 Modality Extensibility

• New modalities can be added without architectural changes: the planner emits modality-agnostic slots, and new APIs can be connected through thin adapters.Source: §A.5 · p. 13

B.1 Instruction-Flow Correction in Practice

• Correction operates on textual prompts before media realization, allowing detection of redundancy or topical drift without inspecting rendered media.The authors leave a feedback loop conditioned on rendered media to future work.Source: §B.1 · pp. 13, 15

B.2 Why UniData Outperforms a Vanilla Pipeline

• UniData pairs GPT-4o with external modality-specific tools, and its higher scores than Self-Instruct under the reported comparison are attributed to event expansion and flow correction.Table 12 compares Self-Instruct on GPT-4 with UniData on LLaMA-3 7B; UniData scores higher on all four GPT-guided metrics.Source: §B.2 · p. 15

C.1 Training and Inference

• DEG is fine-tuned on LLaMA-3 using topic–event pairs from UniDataset, while MIG is fine-tuned on four NVIDIA A100 GPUs.Each fine-tuning run takes roughly 40 hours; DEG uses one A100 with 80 GB, while MIG uses four A100s with 80 GB each.Source: §C.1 · p. 15

C.2 Evaluation Prompt and Metrics

• Fifteen evaluators preferred UniData by a substantial margin across all four dimensions in a forced-choice study of 50 samples.GPT-4o scores assess Reasonableness, Clarity, Detail, and Relevance independently.Source: §C.3 · p. 15, Table 11 · p. 15, §C.2 · p. 15

C.4 Out-of-Distribution Performance

• On 100 out-of-distribution events, UniData remains strongest on every metric while producing richer multimodal content and longer interactions.The test events came from domains outside the seed topic pool and were collected through web scraping followed by manual screening.Source: §C.4 · p. 15, Table 12 · p. 16

C.5 Multimodal Distribution across Dialog Position

• UniDataset distributes multimodal content nearly uniformly across five dialogue-position buckets, each carrying roughly 20% of the multimodal mass.The qualitative examples show modalities appearing in early, middle, and late rounds.Source: §C.5 · p. 16, Table 13 · p. 16

C.6 Adaptation to User-Specified Modalities

• User-specified modality steers generation: image-focused inputs yield roughly 15× more image content, while music-focused inputs similarly boost music counts.The comparison is against the general setting.Source: §C.6 · p. 16, Table 14 · p. 16

C.7 Comparison with Recent Instruction-Data Pipelines

• Unlike recent pipelines with narrower assistant-side modality coverage, UniData combines symmetric nine-modality input/output coverage with long interactions averaging 17.5 rounds.The listed recent pipelines use different tasks and protocols, so their published scores are scope comparisons rather than head-to-head numeric results.Source: §C.7 · p. 16

C.8 Additional Qualitative Results

• For the same keyword pair, three independently sampled DEG events vary in content, tone, and target modality.Figure 10 visualizes this output diversity.Source: §C.8 · p. 16, Figure 10 · p. 18
• MIG examples interleave image, music, code, and emoji across multi-round dialogues, illustrating modality breadth and long-horizon coherence.The examples vary in dialogue length and modality mix while preserving topical coherence, including an extended interaction with text, image, code, and emoji.Source: §C.8 · pp. 16–17, Figure 11 · p. 19, Figure 12 · p. 20

D Future Work

• Future work proposes scaling MIG backbones, adding video, 3D, motion, and tabular tools, and aligning generated data with user deployment specifications.The proposed expansions would extend beyond the current nine modalities and target embodied and scientific domains.Source: D Future Work · p. 17