PaperlayerView exact revision on arXiv

Source-linked AI summary

VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks

Tengju Ru, Weitong Lian, Zecong Tang, Lingyi Meng, Haoran Li, Zhejun Cui, Yichen Zhu, Hangshuo Cao, Qi Kang, Yechi Liu, Kaixuan Wang, Yu-Jie Yuan, Chunwei Wang, Yu Zhang, Bo Dai

arXiv:2611.04418v1cs.CV

TL;DR

High-quality multimodal instruction data is difficult and costly to produce, while raw videos are abundant but challenging to convert into task-oriented data. VidUEU-Agent uses a staged, feedback-driven pipeline to construct VidUEU-100K for understanding, editing, and unified tasks. Fine-tuning baseline models with its data brings consistent performance improvements across representative benchmarks.

  • Problem

    High-quality instruction data for multimodal tasks is scarce and costly to annotate, while raw videos are noisy and do not directly provide task-oriented instruction data.

    Source: §1 · p. 1

  • Method

    VidUEU-Agent processes raw videos through perception, keyframe mining, metadata construction, instruction synthesis, and downstream-feedback-driven data iteration.

    Source: §1 · pp. 2–3

  • Results

    Fine-tuning baseline models with VidUEU-Agent data brings consistent performance improvements across representative benchmarks.

    Source: §5 · p. 10

  • Takeaways & Limitations

    VidUEU-Agent produces a 100K-sample dataset spanning understanding, editing, and unified capabilities as a reproducible and extensible foundation for scalable multimodal data construction.

    Source: §1 · p. 3, §5 · p. 10

    VidUEU-Agent does not construct pure text-to-image data, and its 100K-sample scale remains modest compared with web-scale multimodal corpora.

    Source: J Limitations · p. 54

Abstract

from arXiv · show

The advancement of multimodal learning is heavily constrained by the scarcity and high annotation cost of high-quality instruction data. While raw videos offer an abundant and dynamic source of visual knowledge, transforming them into precise instruction data remains highly challenging. To bridge this gap, we present VidUEU-Agent, an automated pipeline that synthesizes instruction data from raw videos for multimodal understanding, editing, and unified tasks. VidUEU-Agent follows a four-stage pipeline: it performs global perception to filter low-quality shots, mines keyframes and multimodal signals, constructs metadata and routes the metadata to synthesize instruction samples. More importantly, our agent framework can autonomously iterate the data distribution based on downstream training results, thereby optimizing data quality, balance, and training effectiveness. Scaling this pipeline, we construct VidUEU-100K, a large-scale dataset of 100K training samples spanning multiple domains and tasks, providing a robust foundation for developing comprehensive multimodal capabilities. Extensive experiments demonstrate that fine-tuning baseline models on VidUEU-100K yields significant and consistent performance gains across diverse representative benchmarks, validating the effectiveness of our proposed agent framework and dataset.

1 Introduction

High-quality multimodal instruction data is costly to produce, and raw videos are noisy, unstructured, and not directly task-oriented. VidUEU-Agent turns videos into instruction data across understanding, editing, and unified tasks, then iterates its data distribution using downstream training results; VidUEU-100K and benchmark experiments demonstrate the approach’s scale and effectiveness.

1 Introduction

  • The paper identifies a lack of scalable, reproducible production of high-quality UMM instruction data and the challenge of converting noisy raw videos into task-oriented data for understanding and generation.Raw videos are abundant but redundant, noisy, and unstructured.Source: §1 · pp. 1–2
  • VidUEU-Agent uses a four-stage pipeline to filter low-quality shots, mine keyframes and multimodal signals, construct metadata, and synthesize instruction samples for understanding, editing, and unified tasks.The framework also autonomously iterates the data distribution based on downstream training results.Source: §1 · pp. 2–3
  • VidUEU-100K contains 100K instruction samples and 20k dense metadata annotations across multimodal capabilities, domains, and tasks.The contribution summary describes 100K diverse samples covering understanding, editing, and unified capabilities.Source: §1 · p. 3
  • Fine-tuning baseline models on VidUEU-100K improves performance across multiple benchmarks and multimodal tasks.The introduction reports experiments across multiple benchmarks without providing specific metric values in these passages.Source: §1 · p. 3

2 Related Work

Prior agent-based data construction uses tools to improve grounding but has not jointly addressed understanding, editing, and unified instruction data. Meanwhile, video offers temporal dynamics that static image-text datasets lack, motivating its use for multimodal instruction data.

2.1 Data Construction based on Agent Systems

  • Existing agent-based construction methods automate tasks such as collecting boxes, rewriting captions, and synthesizing QA, while LLaVA-Video focuses on video-understanding QA; jointly synthesizing understanding, editing, and unified data remains challenging.Tool use is presented as a way to improve reliability and interpretability of visual reasoning and information extraction.Source: §2.1 · p. 3

2.2 Multimodal Understanding and Generation Datasets

  • Image-text datasets are primarily static, whereas videos contain temporal dynamics and causal interactions that make them a promising medium for instruction data.The passage situates this contrast amid separate progress in multimodal understanding, generation, and unified models.Source: §2.2 · p. 3
  • Figure 2 presents VidUEU-Agent’s sequence from global perception and quality gating through keyframe mining, metadata construction, and instruction synthesis for three task paradigms.The caption names understanding, editing, and unified tasks as the synthesis targets.Source: Figure 2 · p. 4

3 VidUEU: Agent & Dataset

VidUEU-Agent converts raw videos into instruction data through staged perception, keyframe and signal processing, metadata construction, instruction synthesis, and feedback-driven data refinement. The resulting VidUEU-100K dataset draws on diverse video sources and organizes samples across capabilities, domains, and task types.

3.1 VidUEU-Agent

  • VidUEU-Agent uses global perception, selected keyframes and filtered multimodal signals, metadata-to-task mapping, quality screening, and downstream-feedback iteration to curate data for understanding, editing, and unified tasks.The framework overview describes the overall workflow and its quality controls.Source: §3.1 · p. 4

3.1.1 Stage 1: Global Perception

  • Shot-, chapter-, and video-level summaries support Gate A, which removes shots that fail informativeness or image-quality rules before later processing.The preprocessing first detects shot boundaries and divides long videos into shot-level clips.Source: §3.1.1 · p. 4

3.1.2 Stage 2: Keyframe Mining

  • For retained shots, fixed-interval frame sampling and Gate B combine tool-based quality statistics with VLM checks; tools then gather targeted signals and scoring guides representative keyframe selection.Keyframes are selected using weighted facility-location greedy selection after candidates are scored and allocated per-shot budgets.Source: §3.1.2 · pp. 4–5

3.1.3 Stage 3: Metadata Construction

  • Metadata construction filters keyframe-bound signals, generates and verifies transitions and a clip summary, then binds these elements into a metadata list.The generated text is checked for quality and factuality against the signals.Source: §3.1.3 · p. 5

3.1.4 Stage 4: Instruction Data Construction

  • Gate C scores metadata for information content, coherence, and quality, routes suitable entries to one or more task types, and uses signal-ID citations in instruction generation for traceability.Unsuitable metadata is removed before instruction templates are matched to the retained metadata and task list.Source: §3.1.4 · p. 5

3.1.5 Autonomous Data Iteration

  • The agent trains on 30K samples per iteration, evaluates representative benchmarks, and updates target distributions across capabilities, domains, and task types using the results.It adjusts video-type sampling, keyframe budgets and weights, task routing, and quota-based sampling; the final dataset follows Iteration 4’s target distribution.Source: §3.1.5 · p. 6

3.2 VidUEU-100K

  • VidUEU-100K contains 100K instruction-tuning samples—40K understanding, 30K editing, and 30K unified—plus 20k fine-grained metadata.The dataset is constructed using DeepSeek-V3 as the primary agent LLM and Qwen3.5-9B as the VLM.Source: §3.2 · p. 6

3.2.1 Data Collection & Sources

  • The dataset uses 8,268 raw videos from web and open datasets, including general video sources and specialized collections for domain coverage.The pipeline derives approximately 8,505 semantic segments from the source videos for dataset instance generation.Source: §3.2.1 · p. 6

4 Experiment

Fine-tuning on VidUEU-100K improves results across understanding, editing, and unified-task benchmarks. The experiments also examine iterative data refinement, case-study behavior, and the effects of individual pipeline components.

4.1 Experimental Setup

  • The experiments fine-tune InternVL3-8B and Bagel-7B-MoT on capability-specific subsets and evaluate them on representative multimodal benchmarks.The cited setup passage names MMT-Bench, MME, LLaVA-Bench, MMDU, GEdit-Bench-En, and WEAVEbench.Source: §4.1 · p. 6

4.2 Results on Understanding Capabilities

  • InternVL3-8B reaches 2369.3 on MME and 61.1 on MMDU, while Bagel reaches its best MMT-Bench score of 0.651 after fine-tuning.Fine-tuning improves all reported understanding scores for Bagel; MMDU improvements are associated with mining semantically related keyframes and modeling their relations in instruction data.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 Results on Editing Capabilities

  • Bagel fine-tuning improves all reported GEdit-Bench metrics, reaching 7.565 on G_SC, 7.328 on G_PQ, and 6.951 on G_O, best among compared models.The results cover edit correctness and output quality rather than only one aspect of editing.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 Results on Unified Capabilities

  • Bagel’s WEAVEbench Overall score rises from 0.446 to 0.501 after unified-subset fine-tuning, with scores of 0.480 on Logic and 0.419 on Game.The reported gains on Logic and Game are consistent with temporal supervision over structured transitions; the Creation gain is described as modest.Source: §4.4 · pp. 7–8, Table 3 · p. 8

4.5 Case Study

  • Long interleaved image–text chains remain challenging because early errors may propagate, despite better early-turn edit execution and context use.The case study also reports improved evidence reliance for understanding and more realistic target edits that preserve irrelevant regions.Source: §4.5 · p. 8

4.6 Results on Autonomous Data Iteration

  • Repeated feedback-driven refinement produces an upward trend, with more consistent gains across MMDU, GEdit, and WEAVE from Iteration 3 to Iteration 4; Final further improves on Iteration 4.Iterations 1–4 each use 30K samples, while Final scales the Iteration 4 target distribution to 100K samples.Source: §4.6 · p. 8, Table 4 · p. 8

4.7 Ablation Study

  • Removing any major component degrades performance; Gate B removal particularly hurts editing and unified tasks, while the no-agent variant drops the most.The reported ablations also find degradation when removing score-similarity sampling, Gate C metadata filtering, or the consistency hard rule.Source: Table 5 · p. 8, §4.7 · p. 9

5 Conclusion

VidUEU-Agent addresses the challenge of producing high-quality multimodal instruction data from raw videos through a four-stage pipeline and downstream-feedback iteration. The paper constructs VidUEU-100K and reports consistent benchmark improvements from fine-tuning baseline models on its data.

5 Conclusion

  • VidUEU-Agent uses global perception, keyframe mining, metadata construction, and instruction-data construction, then refines data distribution using downstream feedback.VidUEU-100K contains 100K instruction samples spanning understanding, editing, and unified capabilities; experiments report consistent improvements across representative benchmarks.Source: §5 · pp. 9–10

A.1 Stage 1: Global Perception

  • Stage 1 materials include prompts for shot-level summarization, chapter construction and summaries, video-level summaries, and Gate A shot admission.The referenced prompts appear in Figures 5–7.Source: §A.1 · p. 19, Figure 5 · p. 19, Figure 6 · p. 20

A.2 Stage 2: Keyframe Mining

  • Stage 2 materials cover hierarchical tool planning, tool-signal verbalization, frame-value scoring, and quota allocation with weighted facility-location selection.These prompts and rules are referenced in Figures 9–12.Source: §A.2 · p. 19, Figure 9 · p. 22, Figure 10 · p. 23

A.3 Stage 3: Metadata Construction

A.4 Stage 4: Instruction Data Construction

  • Stage 4 materials include prompts for task routing and instruction generation.The appendix also labels a task-generation prompt section and an understanding-generation section; Figures 14 and 15 show the prompts.Source: §A.4 · pp. 19, 28, Figure 14 · p. 27, Figure 15 · p. 28

A.5 LLM–VLM Design and Model Selection

The appendix reports that metadata-driven LLM design is more efficient and scores higher in quality than a direct VLM alternative, while the smaller VLM offers a better measured efficiency–quality balance. It also describes auxiliary tools and the fields contained in the metadata representation.

A.5 LLM–VLM Design and Model Selection

  • The metadata-driven LLM design scores 4.18 versus 2.72 in quality and produces 12.4 versus 5.6 samples/min compared with the direct VLM alternative.Both settings use the same input data, prompt templates, generation parameters, and evaluation protocol, and each produces 200 samples.Source: §A.5 · p. 29, Table 6 · p. 29
  • Qwen3.5-9B is selected over Qwen3.5-27B: quality is 4.16 versus 4.21, while throughput is 12.4 versus 4.1 samples/min/GPU.The passage describes the quality gain from the larger model as marginal and the throughput reduction as the reason for choosing Qwen3.5-9B.Source: §A.5 · p. 29, Table 7 · p. 29

B.1 Tool Pool and Output Types

  • The tool pool supplements frame-level semantic evidence when visual captions are insufficient, using OCR, ASR, object detection, optical flow, and visual parsing tools.The named tools are PaddleOCR, Whisper ASR, Ultralytics YOLOv8, RAFT optical flow, and OmniParser.Source: §B.1 · p. 29, Table 8 · p. 30

B.2 Tool Evidence Verbalization and Scoring Rules

  • The passage identifies Table 9 as summarizing stage-2 rules for verbalizing tool signals and scoring each frame.Source: §B.2 · p. 29

C Metadata Schema Definition

  • The metadata schema binds video segments, chapters, shots, keyframes, and transitions, including timestamps, summaries, quality judgments, tool signals, and transition details.Keyframes carry captions and value scores; transitions include frame-pair ordering and a reliability estimate.Source: C Metadata Schema Definition · p. 29, Figure 16 · p. 30

D Instruction Templates and Examples

The appendix specifies instruction templates spanning visual understanding, editing, and temporally structured generation, alongside examples and frame-selection procedures. The templates ground responses in image evidence and use generated states and candidate selection for several sequence tasks.

D.1 Instruction-Data Synthesis Templates

  • Fine-Grained Perception, Visual Reasoning, and Multi-Image Logic cover single-image detail recognition, cue-based inference, and evidence-grounded cross-image comparison.Their answer formats range from yes/no or multiple choice to open-ended QA and ordered multi-turn dialogue.Source: §D.1 · pp. 30, 32
  • Editing and Grounded Edit-to-Select Generation pair localized image changes with generated targets, while preserving stable scene content and selecting a matching candidate.The editing instruction is constrained to one or two small, controllable edits.Source: §D.1 · p. 32
  • Interleaved and sequence templates combine visual understanding or recall with generated future states and candidate selection; gap filling infers a missing state between endpoints.The instructional sequential template additionally generates future-state instructions and visual checks for each step.Source: §D.1 · pp. 33–35

D.2 Examples

  • The examples illustrate perception, reasoning, multi-image logic, image editing, grounded edit-to-select, and interleaved understanding–recall–generation–selection.They also include stepwise sequence generation, middle-step gap filling, and future-state sequence generation.Source: Figure 17 · p. 36, Figure 18 · p. 37, Figure 19 · p. 37
  • Additional examples show stepwise action sequences, intermediate-frame completion, and future visual steps in a natural blooming process.The section states that it provides one example for each template.Source: §D.2 · p. 36, Figure 23 · p. 41, Figure 24 · p. 42

E.1 Gate A Prompt; E.2 Gate B Rules & Prompt

  • The appendix includes a shot-admission prompt and describes a two-stage frame-quality filter using OpenCV statistics followed by VLM inspection.The fast stage rejects obvious blur, extreme exposure, low contrast, or low informativeness; the VLM checks higher-level visual usability.Source: §E.2 · pp. 44–45, Figure 26 · p. 44

E.3 Facility-location Greedy Selection

  • Weighted facility-location greedy selection chooses representative frames within a shot under a quota, weighting candidate quality and similarity to selected frames.Selection adds the frame with the largest marginal gain until the quota is reached or candidates are exhausted.Source: §E.3 · p. 45

E.4 Signal Filtering Prompt

  • A separate appendix subsection points to the detailed signal-filtering prompt.The prompt itself is shown in Figure 28.Source: §E.4 · p. 45

E.5 Gate C

Gate C filters metadata for quality and assigns suitable task types. The taxonomy organizes the dataset by capability, domain, and task type.

E.5 Gate C

  • Gate C scores metadata for information content, logical coherence, and overall quality, removes unsuitable entries, and routes retained metadata to one or more task types.A 5,000-entry sample had a 37.4% overall rejection rate; cited reasons include unsupported content, noisy signals, generic descriptions, repetitive imagery, and inaccurate transitions.Source: §E.5 · pp. 45, 49, Figure 27 · p. 46, Figure 28 · p. 47

F Dataset Taxonomy Details

  • The taxonomy analyzes VidUEU-100K along three dimensions: capability, domain, and task type.The section provides detailed definitions for these categories.Source: F Dataset Taxonomy Details · p. 49

F.1 Capability Categories

  • The three capability categories are understanding, editing, and unified interleaved text–image tasks.Understanding covers visual comprehension and reasoning; editing modifies or completes images while preserving irrelevant details; unified tasks combine understanding and generation in one stream.Source: §F.1 · p. 49

F.2 Domain Categories

  • The dataset uses five thematic domains and seeks diversity within each to mitigate representation bias and support broad generalization.The domains are technology and tutorial, sports and action, daily life and human activities, nature and scenery, and gaming and ACG.Source: §F.2 · pp. 49–50

F.3 Task Categories

  • The agent adaptively generates nine task types, assigned across understanding, editing, and unified capability paradigms.The types include fine-grained perception, visual reasoning, multi-image logic, editing, and several generate-and-select or gap-filling tasks.Source: §F.3 · p. 50

F.4 Data Statistics

VidUEU contains 100,000 instruction samples derived from 8,268 raw videos. The section also describes a human audit, model-training setups, evaluation settings, and comparison baselines.

F.4 Data Statistics

  • 100,000 samples comprise 40,000 understanding, 30,000 editing, and 30,000 unified samples, derived from 8,268 videos spanning about 224.2 hours and 8,505 semantic segments.The capability, domain, and task-type distributions are shown in Figure 30.Source: §F.4 · p. 50

F.5 Intrinsic Data-Quality Audit

  • The human audit randomly sampled 800 instances, with 200 per quality dimension, and used two independent annotators scoring each on a 0–5 scale.The first three dimensions were assessed at metadata level, while editing validity was assessed at the instruction level; Table 11 reports means and Cohen’s κ.Source: §F.5 · p. 50, Table 11 · p. 51

G.1 Training Details

  • InternVL3-8B was trained on the understanding subset for two epochs with a frozen vision encoder and LoRA; Bagel-7B-MoT used one epoch and LoRA, with generation heads additionally fine-tuned for editing and unified tasks.The passages also specify optimizer, learning-rate, hardware, and other training settings.Source: §G.1 · p. 51

G.2 Evaluation Details

  • Fine-tuned models are evaluated in understanding, editing, and unified understanding-and-generation settings using each benchmark’s official protocol.The compared baselines span representative understanding, editing, and unified models.Source: §G.2 · p. 51

G.3 Compared Baselines

  • Comparisons include InternVL3 model sizes, Qwen2.5-VL, LLaMA-3.2, Janus-Pro, and TokenFlow-XL, alongside image-editing and unified baselines such as Instruct-Pix2Pix, OmniGen2, Ovis-U1, and Bagel-Zebra.The passage lists additional editing and unified baselines.Source: §G.3 · p. 51

G.4 Evaluation Protocols

The appendix specifies the evaluation judges and variability measurement, then details autonomous data iterations and their training and distribution-feedback protocol.

G.4 Evaluation Protocols

  • Understanding benchmarks use specified VLM judges, GEdit-Bench-En uses GPT-4.1 for its listed metrics, and WEAVEbench uses GPT-4o.The judges are Qwen3.5-35B-A3B and Qwen3-VL-235B-A22B-Instruct for LLaVA-Bench and MMDU, respectively.Source: §G.4 · p. 51

G.5 Evaluation Variability

  • Evaluation variability is measured as standard deviation over three independent judge runs with evaluated model outputs held fixed.Table 12 contains these run-to-run standard deviations; mean scores appear in the main experimental tables.Source: §G.5 · p. 51, Table 12 · p. 52

H.1 Full Results of Autonomous Data Iteration

  • Table 13 reports results for four 30K-sample iterations and the final capability-specific VidUEU-100K training; final joint-training results are reported in Table 4.The supplied passage does not state the table’s result values.Source: §H.1 · p. 52, Table 13 · p. 52

H.2 Training Protocol and Data Distributions

  • Each iteration samples 30K examples according to updated capability, domain, and task proportions, while the final 100K dataset scales the distribution from Iteration 4.The agent receives training and benchmark feedback, current distributions, compressed history, and optionally human-audit inputs; capability proportions take priority, with domain and task deviations limited to one percentage point.Source: §H.2 · p. 52

H.3 Feedback-Free Evaluation

Across feedback-free evaluation, full ablations, baseline-dataset comparisons, and broader impacts, the passages report an upward trend, performance degradation when major components are removed, stronger results across evaluated settings, and potential inherited source-video risks.

H.3 Feedback-Free Evaluation

  • MMBench, MMStar, and VSI-Bench, excluded from autonomous feedback, show an overall upward trend across iterations.The excluded benchmarks assess generalization beyond the feedback benchmarks.Source: §H.3 · p. 52, Table 17 · p. 53

H.4 Full Ablation Study

  • Removing any major VidUEU-Agent component degrades performance, supporting the full-pipeline design.Ablated variants use 10K samples per capability, while the 100K Final models serve as the reference; the component comparisons are indicative, not strictly budget-matched.Source: §H.4 · p. 53, Table 18 · p. 54

H.5 Comparison with Baseline Datasets

  • VidUEU-100K achieves the best results across all evaluated settings against three task-specific baseline datasets using Bagel under the same training setting.The baselines are LLaVA Visual Instruct 150K for understanding, InstructPix2Pix for editing, and Zebra-CoT for unified tasks.Source: §H.5 · p. 53, Table 19 · p. 54

I Broader Impacts

  • Source videos may pass social biases, privacy-sensitive content, or copyright risks to the dataset and downstream models despite careful human screening.The authors recommend careful data filtering, license review, privacy protection, and safeguards against malicious use.Source: I Broader Impacts · p. 54

J Limitations

VidUEU-Agent omits explicit pure text-to-image data construction and scales VidUEU-100K to 100K samples, which remains modest compared with web-scale multimodal corpora.

J Limitations

  • VidUEU-Agent does not explicitly construct pure text-to-image generation data.The authors favor directly curating image data for this task because it mainly requires isolated text–image alignment, making image curation more suitable and less costly than using videos as the primary source.Source: J Limitations · p. 54
  • VidUEU-100K contains 100K instruction samples, a scale the authors describe as modest compared with web-scale multimodal corpora.The authors attribute this scale to limited computational and annotation resources.Source: J Limitations · p. 54
  • The authors suggest a larger-scale version may improve data diversity, long-tail coverage, and multimodal training stability.Source: J Limitations · p. 54

Related papers

Loading 2611.04418v1…