Source-linked AI summary
Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructions
Yunheng Li, Hengrui Zhang, Meng-Hao Guo, Wenzhao Gao, Shaoyong Jia, Shaohui Jiao, Qibin Hou, Ming-Ming Cheng
TL;DR
Existing video-instruction data often represents complex audiovisual content as incomplete, unstructured captions, limiting reliable fine-grained supervision. The paper introduces ASID-1M, the ASID-Verify curation pipeline, and ASID-Captioner trained with progressive supervised fine-tuning. Across seven benchmarks, ASID-Captioner improves fine-grained caption quality and instruction following, with performance competitive with Gemini-3-Pro.
Problem
Existing datasets provide incomplete fine-grained supervision, unreliable multi-attribute annotations, limited systematic verification, and restricted openness for reproducibility.
Method
The paper constructs open-source ASID-1M with single- and multi-attribute instructions, uses ASID-Verify for annotation, verification, and refinement, and trains ASID-Captioner with progressive supervised fine-tuning.
Results
Across captioning, instruction following, caption-grounded QA, and temporal grounding evaluations, ASID-Captioner achieves consistent gains over strong open-source models and remains competitive with Gemini-3-Pro.
Takeaways & Limitations
Structured and quality-verified audiovisual supervision supports higher-quality captions and improved attribute-level instruction following.
Takeaways & Limitations
Residual noise may remain, evaluations partly depend on proprietary-model judges or solvers, and ASID-1M focuses on short-form videos, leaving longer-video and cross-domain generalization for further validation.
Abstract
from arXiv · showhide
Universal video understanding requires modeling fine-grained visual and audio information over time in diverse real-world scenarios. However, the performance of existing models is primarily constrained by video-instruction data that represents complex audiovisual content as single, incomplete descriptions, lacking fine-grained organization and reliable annotation. To address this, we introduce: (i) ASID-1M, an open-source collection of one million structured, fine-grained audiovisual instruction annotations with single- and multi-attribute supervision; (ii) ASID-Verify, a scalable data curation pipeline for annotation, with automatic verification and refinement that enforces semantic and temporal consistency between descriptions and the corresponding audiovisual content; and (iii) ASID-Captioner, a video understanding model trained via Supervised Fine-Tuning (SFT) on the ASID-1M. Experiments across seven benchmarks covering audiovisual captioning, attribute-wise captioning, caption-based QA, and caption-based temporal grounding show that ASID-Captioner improves fine-grained caption quality while reducing hallucinations and improving instruction following. It achieves state-of-the-art performance among open-source models and is competitive with Gemini-3-Pro.
1 Introduction
Existing video-instruction data often compresses complex audiovisual content into incomplete, unreliable captions, limiting fine-grained learning. The paper introduces open, attribute-structured supervision, verification, and a model that improves fine-grained captioning and instruction following.
- Motivation: Existing caption-based supervision often omits fine-grained information, constraining models because supervised fine-tuning learns directly from what captions explicitly provide.Reinforcement learning can adjust generation preferences but cannot reliably reconstruct information absent from training supervision.
- Motivation: Detailed multi-attribute annotation can introduce incorrect and hallucinated content, making granularity alone insufficient for reliable supervision.Prior datasets also generally lack systematic verification because fine-grained validation is costly and difficult to scale.
- Contributions: ASID-1M provides open-source, million-scale audiovisual instructions decomposed into complementary single- and multi-attribute descriptions.The dataset replaces single-caption modeling with attribute-structured instructions for more explicit fine-grained learning.
- Contributions: ASID-Verify performs automatic annotation, verification, and refinement to enforce semantic and temporal alignment between instructions and audiovisual content.The pipeline is designed to improve reliability and consistency of fine-grained supervision.
- Results: ASID-Captioner, trained on ASID-1M, achieves consistent gains in fine-grained semantic coverage and attribute-level instruction following over strong open-source models.Evaluation spans seven benchmarks, and the 7B variant remains competitive with Gemini-3-Pro on several evaluations.
2 Related Work
Audiovisual multimodal models unify visual, audio, and language understanding, but existing instruction datasets generally provide coarse or monolithic supervision. Recent datasets improve conversational coverage while retaining limited fine-grained temporal and semantic annotation.
- Audiovisual Multimodal Models: Audiovisual MLLMs jointly model visual representations, acoustic signals, and natural language for instruction following, reasoning, and video interaction.Most existing methods use supervised fine-tuning on instruction-style video-language data and generate single monolithic responses.
- Video Instruction Datasets: Early video-text datasets primarily provide coarse video-level descriptions that support broad multimodal alignment.Their supervision does not directly represent fine-grained temporal structure or semantic composition.
- Video Instruction Datasets: More recent conversational video datasets increase descriptive coverage and interaction flexibility but still provide limited fine-grained temporal and semantic annotation.This limitation motivates structured supervision for universal video understanding.
3 ASID-1M and ASID-Verify Pipeline
ASID-Verify constructs ASID-1M through staged multi-source annotation, ensembling, audiovisual verification, and targeted refinement. The resulting dataset contains verified attribute-specific and all-attribute captions, with manual inspection indicating high reliability.
- Pipeline Overview: ASID-Verify uses a staged, scalable pipeline that integrates annotation ensembling, validation, and targeted refinement instead of single-pass annotation.The pipeline is designed to address missing structured fine-grained annotation and systematic verification.
- S1: Multi-Source Annotation Generation: Initial annotations combine complementary audiovisual sources, including open-source models and ASR-aligned spoken content, to broaden coverage.The initial video pool contains approximately 125K short-form or YouTube-subset videos.
- S2: Caption Ensembling and Verification: Caption ensembling synthesizes multiple candidate captions into a unified all-attribute draft while retaining complementary details and resolving obvious redundancy or conflicts.The draft is subsequently checked against the original audiovisual evidence because fusion can introduce missing details or incorrect combinations.
- S2: Caption Ensembling and Verification: Verification grounds speech descriptions through ASR alignment and checks timestamp consistency across semantic attributes before selectively removing inconsistent content.This produces captions more comprehensive than individual source candidates while preserving validated content.
- S3: Attribute-Wise Verification and Refinement: Attribute-wise verification independently evaluates predefined attributes for errors, missing content, fluency, redundancy, and ambiguous references, enabling targeted refinement.Selective editing affects only problematic attributes while preserving the remaining caption content.
- Final Dataset: 121K videos form the final ASID-1M, each paired with eight verified single-attribute captions and one all-attribute caption; manual inspection found over 98% reliable refined captions.Remaining issues were limited to minor timestamp deviations within 2 seconds in the inspected subset.
- Stage-Wise Analysis: Stage-wise analysis evaluates annotation quality through No-Error, No-Missing, and Clean rates, alongside downstream effectiveness after training on video captions.The analysis shows that single models miss substantial fine-grained information and that specialized models capture complementary content.
4 Progressive Attribute Learning
Progressive attribute learning expands supervision from individual attributes to joint, long-context descriptions, then enables controllable attribute selection at inference.
- Training stages: The three-stage scheme progresses from single-attribute supervision to all-attribute learning on short and then long video clips.Stage 3 uses clips up to 3 minutes while keeping the Stage 2 objective unchanged.
- Training stages: Stage 1 pairs each video and selected attribute with an attribute-specific caption, enforcing attribute-specific learning through negative log-likelihood optimization.The training data consists of samples (v, a, y_a) from D1.
- Training stages: Stage 2 jointly models the full attribute set and a joint caption under a short temporal context.This stage shifts learning from attribute-wise modeling to joint modeling over all attributes.
- Training stages: Stage 3 exposes the model to extended temporal context by training on long clips under all-attribute supervision.The long clips extend to 3 minutes, using a distinct data distribution D3.
- Inference: At inference, users select any attribute subset A′, allowing caption content to be controlled without retraining or architectural changes.The generated caption is conditioned on the selected attributes.
5 Experiments
Across seven complementary evaluations, ASID-Captioner improves audiovisual, attribute-wise, generation-oriented, QA, temporal-grounding, and instruction-following performance. Ablations link these gains to structured multi-attribute supervision and progressive training.
- Audiovisual captioning: ASID-Captioner balances completeness and reliability across Video-SALMONN-2 and UGC-VideoCap, with strong audio, visual, and detailed-caption performance.It reduces missing content while maintaining controlled hallucination levels and outperforms prior open-source omni models on UGC-VideoCap.
- Attribute-wise visual captioning: ASID-Captioner performs consistently across VDC’s visual attributes, with particular advantages for camera and main-object details.It is competitive with Gemini models and surpasses AVoCaDO in averaged attribute-wise performance.
- Text-to-video generation caption: ASID-Captioner achieves strong VidCapBench-AE performance, with especially pronounced gains in video content and motion.The benchmark evaluates captions for video aesthetics, content, motion, and physical laws using precision and coverage measures.
- Caption-grounded QA: Captions from ASID-Captioner produce the strongest caption-only QA results on Daily-Omni and World-Sense, including for both 3B and 7B variants.The advantage is more pronounced on World-Sense, which requires reasoning over entities, interactions, and speech.
- Caption-based temporal grounding: Caption-only temporal grounding shows clear improvements at higher IoU thresholds, indicating preservation of temporally precise event information.A pretrained Gemini-2.5-Pro predicts event start and end times from captions alone.
- Supervision ablation: Multi-attribute supervision substantially reduces missing content and improves detailed caption quality compared with raw or non-attribute annotations.The evaluation also notes that finer descriptions can cause relevant but unannotated details, such as on-screen text, to be penalized as hallucinations.
- Stage-wise training ablation: Stage-wise ablations show that single-attribute training supports basic grounding, while adding short-context all-attribute training improves missing-content and detailed-caption metrics.All ablations use 20K randomly sampled instances and Qwen2.5-Omni-3B under the same optimization settings.
- Attribute-level instruction following: ASID-Captioner follows requested attribute combinations more accurately and generalizes to unseen combinations as the number of requested attributes increases.Existing captioning models show low accuracy even for prompts requesting one or two attributes.
6 Conclusions
The paper presents ASID-1M and ASID-Verify for reliable, attribute-structured audiovisual supervision, then trains ASID-Captioner on this data. Across captioning, instruction-following, caption-grounded QA, and temporal-grounding evaluations, it reports consistent gains over strong open-source models and competitiveness with Gemini-3-Pro.
- ASID-1M provides composable single- and multi-attribute audiovisual instructions, while ASID-Verify produces reliable attribute-structured supervision.The pipeline combines multi-source annotation, consistency verification against audiovisual content, and targeted attribute-wise refinement.
- ASID-Captioner produces higher-quality audiovisual captions with improved attribute-level instruction following after progressive supervised fine-tuning on ASID-1M.
- Across captioning, instruction following, caption-grounded QA, and temporal-grounding evaluations, ASID-Captioner achieves consistent gains over strong open-source models.
Limitations
The paper identifies residual annotation noise, evaluator dependence, and short-form-video scope as limitations requiring caution in interpreting results and generalization.
- Residual annotation noise may remain, particularly for ASR-dependent speech content and fine-grained timestamps.
- Some evaluations rely on proprietary models as judges or downstream solvers, which may introduce evaluator bias and make absolute scores judge-dependent.
- ASID-1M focuses on short-form videos, so generalization to longer videos or different domains requires further validation.
A Implementation Details
ASID-Captioner uses staged supervised fine-tuning with fixed optimization settings and computational controls; Table 10 summarizes the training-stage configuration.
- All experiments use Qwen2.5-Omni and supervised fine-tuning, with AdamW, a 2 × 10^-5 learning rate, and global batch size 128.
- A fixed pixel budget per sample controls computational cost, with a reduced total pixel budget in Stage 3.
- DeepSpeed with ZeRO-2 is adopted for optimization efficiency and memory scalability in the training stages described.
- Table 10 summarizes the training-stage configuration of ASID-Captioner.
B Annotation Datasets
The annotation and evaluation materials span diverse short-form audiovisual videos and benchmarks covering caption quality, structured visual description, audiovisual QA, and temporal grounding.
- LLaVA-Video-178K contains diverse untrimmed YouTube clips from 5 seconds to 3 minutes, emphasizing temporal diversity and dynamic scenes; approximately 119K videos are sampled for annotation.
- FineVideo contains approximately 43K videos totaling about 3.4K hours, emphasizing dynamic visual content and dense spoken language; approximately 16K short-form videos are sampled.
- video-SALMONN-2 tests caption quality on 483 videos across 14 domains using missing, incorrect, and hallucinated event errors.
- UGC-VideoCap evaluates 1,000 TikTok videos with judge scores for visual content, audio content, and descriptive detail.
- VDC evaluates structured visual captions across camera, short summary, background, main object, and detailed description using aspect-level accuracy and judge scores.
- VidCapBench-AE reformulates caption evaluation as aspect-specific QA covering aesthetics, content, motion, and physical laws, with accuracy, precision, coverage, and conciseness.
- Daily-Omni and WorldSense measure how generated captions support audiovisual question answering, while Charades-STA evaluates caption-based temporal localization using mean IoU and recall.
D Additional Experiments
The ablation examines whether Stage 1 attribute-wise representation learning improves audiovisual captioning. Introducing this supervision reduces missing content and total error, while its effect on hallucination is limited.
- Training Stage Ablation: Stage 1 attribute-wise supervision consistently reduces missing content on video-SALMONN-2.The comparison evaluates models trained with and without the attribute-wise representation learning stage.
- Training Stage Ablation: Stage 1 supervision improves overall caption reliability and total error on video-SALMONN-2.The reported total-error improvement indicates more accurate semantic grounding from early attribute-wise supervision.
- Training Stage Ablation: The effect of Stage 1 supervision on hallucination is limited.
- Training Stage Ablation: Table 11 reports the ablation of Stage 1 training supervision for audiovisual captioning performance.
E Attribute Coverage Analysis
The analysis organizes fine-grained video semantics hierarchically and illustrates caption generation across varied real-world scenarios. Its annotation and quality-control prompts target detailed, temporally aligned, verifiable audiovisual descriptions.
- Attribute Taxonomy: The attribute taxonomy has three levels, moving from coarse to fine-grained semantic categories.The first two levels are manually defined as the annotation schema, while the finest level is derived from completed annotations.
- Attribute Taxonomy: The annotation schema covers semantic dimensions including scene, action, object, and speech.
- Caption Visualization: Captions generated across landscapes, urban scenes, narratives, interactions, and sports are described as fine-grained, temporally coherent, and well-grounded.These visualizations span diverse real-world scenarios and instruction settings.
- Data Curation Prompts: The data pipeline generates timestamped captions from subtitle-annotated clips and ensembles multi-source descriptions into one coherent narrative.
- Caption Verification: Quality control verifies and completes captions by identifying visual and speech deficiencies, then audits attributes for verifiable errors and missing items.The auditing prompt restricts evaluation to items explicitly listed in each dimension’s VERIFY scope.
- Caption-Based Evaluation: The evaluation prompts support multiple-choice question answering and temporal grounding based on textual video captions.