PaperlayerView exact revision on arXiv

Source-linked AI summary

ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shorts

Yuying Ge, Yixiao Ge, Chen Li, Teng Wang, Junfu Pu, Yizhuo Li, Lu Qiu, Jin Ma, Lisheng Duan, Xinyu Zuo, Jinwen Luo, Weibo Gu, Zexuan Li, Xiaojing Zhang, Yangyu Tao, Han Hu, Di Wang, Ying Shan

arXiv:2507.20939v1cs.CV

TL;DR

Real-world shorts’ dense visuals, rich audio, and rapid pacing pose a challenge for multimodal models that struggle to deliver the required in-depth comprehension. ARC-Hunyuan-Video combines synchronized visual, audio, and textual processing with temporally precise video understanding. It achieves the highest ShortVid-Bench accuracy and outperforms baselines in temporal grounding, with reported gains in product retrieval engagement.

  • Problem

    Real-world shorts combine dense visual elements, information-rich audio, and rapid pacing, while current multimodal models struggle to provide the in-depth comprehension needed for these videos.

    Source: §1 · pp. 1–2

  • Method

    ARC-Hunyuan-Video processes visual, audio, and textual inputs for temporally precise video comprehension, using timestamped frames, synchronized audio-visual features, and staged training.

    Source: §1 · p. 3

  • Results

    ARC-Hunyuan-Video achieves the highest accuracy on ShortVid-Bench and outperforms all baselines in temporal video grounding.

    Source: §4.2.2 · p. 16

  • Takeaways & Limitations

    In product video retrieval, the brief-summary application increased retrieval CTR by 5.88%.

    Source: §4.3.3 · p. 20

Abstract

from arXiv · show

Real-world user-generated short videos, especially those distributed on platforms such as WeChat Channel and TikTok, dominate the mobile internet. However, current large multimodal models lack essential temporally-structured, detailed, and in-depth video comprehension capabilities, which are the cornerstone of effective video search and recommendation, as well as emerging video applications. Understanding real-world shorts is actually challenging due to their complex visual elements, high information density in both visuals and audio, and fast pacing that focuses on emotional expression and viewpoint delivery. This requires advanced reasoning to effectively integrate multimodal information, including visual, audio, and text. In this work, we introduce ARC-Hunyuan-Video, a multimodal model that processes visual, audio, and textual signals from raw video inputs end-to-end for structured comprehension. The model is capable of multi-granularity timestamped video captioning and summarization, open-ended video question answering, temporal video grounding, and video reasoning. Leveraging high-quality data from an automated annotation pipeline, our compact 7B-parameter model is trained through a comprehensive regimen: pre-training, instruction fine-tuning, cold start, reinforcement learning (RL) post-training, and final instruction fine-tuning. Quantitative evaluations on our introduced benchmark ShortVid-Bench and qualitative comparisons demonstrate its strong performance in real-world video comprehension, and it supports zero-shot or fine-tuning with a few samples for diverse downstream applications. The real-world production deployment of our model has yielded tangible and measurable improvements in user engagement and satisfaction, a success supported by its remarkable efficiency, with stress tests indicating an inference time of just 10 seconds for a one-minute video on H20 GPU.

1 Introduction

Real-world shorts combine dense visual and audio information with rapid pacing, while current multimodal models struggle to provide the in-depth comprehension these videos require. ARC-Hunyuan-Video targets this gap with temporally structured multimodal understanding and reports strong benchmark and deployment outcomes.

1 Introduction

  • Short videos’ dense visuals, information-rich audio, and rapid pacing challenge current multimodal models, which fall short of the comprehension needed for real-world applications.Structured comprehension is motivated by applications including search, recommendation, and emerging intelligent video services.Source: §1 · pp. 1–2
  • The model supports multi-granularity timestamped captioning, summarization, temporal grounding, and open-ended question answering by reasoning over visual and audio cues.Timestamped captions pair a time span with its corresponding description.Source: Figure 1 · p. 2
  • Structured Video Comprehension decomposes real-world shorts into events and narrative elements with temporal precision across timestamped captioning, summaries, video reasoning, and temporal grounding.The model integrates visual, audio, and textual inputs to address what happens, when it happens, why it matters, and creator intent.Source: §1 · p. 3
  • The model adds an audio encoder with fine-grained visual-audio synchronization and overlays timestamps on frames, then trains through five stages including RL post-training and final instruction tuning.The training uses an automated annotation pipeline, high-quality human-annotated data, and rejection-sampled trajectories; its RL strategy uses objective questions such as multiple-choice and temporal-grounding tasks.Source: §1 · p. 3
  • ARC-Hunyuan-Video, a compact 7B-parameter model, achieves exceptional performance in real-world video understanding on the human-annotated ShortVid-Bench and excels in temporal video grounding benchmarks.Real-world deployment reportedly improved user engagement and satisfaction; stress tests measured 10 seconds per one-minute video on an NVIDIA H20 GPU, averaging 500 tokens, with inference accelerated by vLLM.Source: §1 · p. 3
  • The authors open-source the model checkpoint, API, and inference code to facilitate further research and application.Source: §1 · p. 3

2 Related Work

Prior short-video and audio-visual systems leave gaps in raw-audio integration or in handling the pace and information density of user-generated shorts. ARC-Hunyuan-Video is presented as a compact multimodal model for structured comprehension of these videos.

2 Related Work

  • Short videos’ dense visuals, rich audio, and rapid narrative pacing call for joint visual, audio, and text modeling with reasoning about events, temporal relations, and intent.Source: §2 · p. 3
  • Keye-VL-8B relies on ASR transcripts rather than directly integrating raw audio, while general-scenario audio-visual LLMs often struggle with fast-paced, information-rich shorts.The paper notes that transcript-based processing can discard non-speech cues and create temporal misalignment.Source: §2 · pp. 3–4
  • ARC-Hunyuan-Video is a compact 7B-parameter multimodal model that targets structured comprehension of real-world shorts through synchronized audio-visual-text processing and reasoning.Source: §2 · p. 4

3 Method

Figure 2 presents ARC-Hunyuan-Video’s model architecture and multi-stage training regimen, combining aligned visual-audio inputs with temporal cues and staged model training.

3 Method

  • The architecture builds on Hunyuan-7B VLM, adds an audio encoder with fine-grained visual-audio synchronization, and overlays timestamps on visual frames for temporal awareness.Source: Figure 2 · p. 4
  • The training sequence comprises pre-training, instruction fine-tuning, cold-start initialization, RL post-training, and final instruction fine-tuning.The final stage uses high-quality human-annotated data and trajectories selected via rejection sampling.Source: Figure 2 · p. 4
  • Figure 2 depicts temporally aligned multimodal inputs alongside the staged training process.Source: Figure 2 · p. 4

3.1 Model Architecture

The model encodes timestamped visual frames and audio, synchronizes features corresponding to the same time interval, and feeds the resulting multimodal sequence to the LLM.

3.1 Model Architecture

  • The model adds an audio encoder and fine-grained visual-audio synchronization to Hunyuan-7B VLM, while frame timestamps provide explicit temporal awareness.Source: §3.1 · p. 4
  • Frames are sampled at 1 fps, capped at 150 uniformly sampled frames for videos longer than 150 seconds, timestamped in HH:MM:SS, resized to 640×640, and encoded into 112 visual tokens each.Source: §3.1 · p. 4
  • The audio encoder processes raw waveforms in 30-second chunks, producing 1,500 feature tokens per chunk, and an MLP projects audio features to the visual-token dimensionality.For videos longer than 300 seconds, the described design splits audio into 150 segments and truncates each to its initial 2 seconds.Source: §3.1 · pp. 4–5
  • A parameter-free synchronization strategy aligns each sampled frame with audio from the corresponding interval, padding audio tokens to match visual-token counts before adding the features.The resulting synchronized embeddings receive positional encodings before entering the LLM.Source: §3.1 · p. 5

3.2 Pre-training

ARC-Hunyuan-Video builds structured short-video comprehension through an iterative annotation pipeline, diverse pre-training data, and a progressive strategy for integrating audio while preserving visual capabilities.

3.2 Pre-training

  • The bootstrapped pipeline starts with timestamped ASR speech, frame-level MLLM descriptions, and metadata, then feeds initial annotations back through a trained model for final annotation.The pipeline's iterative refinement is intended to improve multimodal integration.Source: Figure 3 · p. 5, §3.2.1 · p. 5

3.2.1 Annotation pipeline

  • The annotation pipeline combines timestamped speech, frame captions and OCR, and video metadata for LLM synthesis, then uses model-generated descriptions to iteratively polish final annotations.The LLM is prompted to produce event descriptions, creator attitude, and audience-interest tags before generating a summary.Source: §3.2.1 · p. 5

3.2.2 Pre-training data

  • Pre-training data spans 4.5M real-world shorts, 0.2M academic videos, 4.7M image-text pairs, 3.2M audio-text pairs, 0.5M grounding instances, and event- and chapter-level captions.The caption data includes 50K event-level samples and chapter captions for 80K in-house videos.Source: §3.2.2 · p. 6

3.2.3 Training Recipe

  • Training first warms up on ASR and image-text data to align audio while retaining visual understanding, then performs full video/audio/text next-token pre-training.During full multimodal pre-training, the ViT and audio encoder are frozen while the MLP adapters and LLM backbone are updated.Source: §3.2.3 · p. 6

3.3 Post-training

Post-training combines instruction tuning, a reasoning-oriented cold start, GRPO on verifiable tasks, and final tuning on human annotations and sampled trajectories. Pilot experiments report gains after GRPO followed by summary fine-tuning, unlike direct supervised fine-tuning or DPO on summaries.

3.3 Post-training

  • Because automated annotations can contain noise, post-training adds a small set of high-quality human-annotated data to improve real-world short-video comprehension.The pilot experiments investigate how to use this data effectively.Source: §3.3 · p. 6

3.3.1 Pilot Experiments

  • GRPO on objective video tasks followed by summary fine-tuning reached 6.99, versus 6.42 pretrained, while direct summary fine-tuning and DPO reached 6.67 and 6.50.The authors attribute the gains to multiple-choice comprehension dimensions and grounding-based temporal awareness; they hypothesize distribution mismatch may explain the weaker direct approaches.Source: §3.3.1 · pp. 6–7

3.3.2 Stage 1: Initial Instruction Fine-tuning

  • Initial instruction fine-tuning uses open-ended and multiple-choice QA, grounding, description and summary, and multi-granular captioning data.The data includes public academic and real-world short-video samples.Source: §3.3.2 · p. 7

3.3.3 Stage 2: Cold Start Initialization for Reinforcement Learning

  • Cold start fine-tunes on 146K reasoning samples, retaining multiple-choice and grounding rationales by answer correctness and IoU filters, respectively.It also covers open-ended QA, summarization, and chapter-level captioning, and serves as the starting policy for reinforcement learning.Source: §3.3.3 · p. 7

3.3.4 Stage 3: Reinforcement Learning with GRPO

  • GRPO uses 100K multiple-choice questions with binary correctness rewards and 35K temporal-grounding instances rewarded by interval IoU.This stage focuses on objective, verifiable reward signals.Source: §3.3.4 · p. 8

3.3.5 Stage 4: Final Instruction Fine-tuning

  • Final instruction fine-tuning combines 25K human-annotated subjective questions with rejection-sampled trajectories: 100K multiple-choice and 50K grounding instances.The sampled examples include chain-of-thought rationales or detailed reasoning traces.Source: §3.3.5 · p. 8

4.1 Qualitative Evaluation

Qualitative examples show ARC-Hunyuan-Video handling cross-modal questions, fine-grained event timing, and thematic interpretation in real-world shorts. Comparisons with baseline models emphasize its audio-visual reasoning and temporal awareness.

4.1.1 Model Capability

  • In a tutorial, the model combines a voltage-tester action with spoken instructions to answer a safety question, and it organizes review details such as specifications, prices, and target users.The examples illustrate information distributed across visuals, on-screen text, and narration.Source: §4.1.1 · p. 8
  • The model segments an outlet-replacement tutorial into timestamped steps and identifies distinct scenes with time ranges in a fast-paced morning-routine montage.The examples include steps from turning off the breaker through testing the installation.Source: §4.1.1 · p. 8, Figure 4 · p. 9
  • The model identifies creative strategies and the thematically resonant final slogan shot in an environmental promotional video.The examples characterize its capability as reasoning about emotional weight and core intent beyond literal description.Source: Figure 5 · p. 10, §4.1.1 · p. 12

4.1.2 Model Comparison

  • Against baselines, ARC-Hunyuan-Video captures a skit's satirical premise through audio-visual fusion and interprets the emotional purpose of an umbrella video beyond its shared visual contrast.The comparisons describe baseline summaries as missing audio context or offering more superficial descriptions.Source: §4.1.2 · p. 12, Figure 7 · p. 13, Figure 8 · p. 14
  • In temporal-grounding examples, the model locates cooking and food-commentary events that baseline predictions miss or time imprecisely.The authors attribute this capability to timestamp awareness and joint visual-audio reasoning.Source: §4.1.2 · p. 12, Figure 9 · p. 15

4.2 Quantitative Evaluation

The evaluation covers ShortVid-Bench, temporal grounding, and general video-understanding benchmarks. ARC-Hunyuan-Video-7B achieves the highest accuracy on ShortVid-Bench, outperforms baselines on temporal grounding, and shows promising results on general benchmarks.

4.2.1 Evaluation Benchmark

  • ShortVid-Bench uses human-curated multiple-choice questions across six dimensions, testing comprehension of context, intent, and narrative through visual and audio cues.Its dimensions are temporal reasoning and localization, affective intent, creator intent, narrative comprehension, humor and meme deconstruction, and creative innovation analysis.Source: §4.2.1 · p. 16
  • Temporal grounding evaluation includes Charades-STA, with 3,720 indoor-activity videos, and ActivityNet, with 17,031 test samples.Source: §4.2.1 · p. 16
  • General video evaluation covers MVBench, multiple-choice VCR-Bench, and Video-Holmes, which focuses on complex reasoning in suspenseful short films.Source: §4.2.1 · p. 16

4.2.2 Evaluation Results

  • ARC-Hunyuan-Video-7B achieves the highest accuracy on ShortVid-Bench and outperforms all baselines in temporal grounding, with promising results on general video benchmarks.The authors attribute grounding performance largely to timestamp overlays that enhance temporal awareness.Source: Table 1 · p. 16, §4.2.2 · p. 16

4.3 Downstream Application

The section evaluates minimal-data supervised fine-tuning for three downstream scenarios and reports improvements over online baselines, alongside production benefits. Detailed summaries remain more difficult than brief summaries.

4.3 Downstream Application

  • The three fine-tuning tasks map brief summaries to retrieval, detailed summaries to tagging, and extended browsing words to recommendation.Brief summaries condense people, location, and events; detailed summaries cover video content and production details; extended browsing words are likely follow-on search terms.Source: §4.3 · p. 16, §4.3.1 · pp. 18–19

4.3.1 Experimental Setup

  • Each task used 1,100 manually annotated samples, split into 1,000 for fine-tuning and 100 for evaluation; annotation cross-validation pass rate exceeded 95%.Evaluation used a 0–2 Pass Rate scale and blind expert comparisons of outputs in the Good vs. Same vs. Bad metric.Source: §4.3.1 · p. 19

4.3.2 Implementation Details

  • Implementation optimized the MLP adapter layers and full LLM backbone at learning rate 1e-5 for three epochs, using rule-description prompts and a two-step reasoning structure for recommendation terms.Source: §4.3.2 · p. 19

4.3.3 Experimental Analysis

  • Across three downstream tasks, supervised fine-tuning yielded higher Pass Rates and dominant GSB win rates than the respective online baselines.The authors report fewer zero scores and more full scores for the fine-tuned model.Source: §4.3.3 · p. 19, Table 2 · p. 20
  • Detailed summaries had more zero scores and fewer full scores than brief summaries, reflecting greater difficulty from stricter rules and greater risk of errors or omissions.Source: §4.3.3 · p. 20
  • Video retrieval deployment raised retrieval CTR 5.88%, landing-page consumption time 5.11%, floating-layer click CTR 7.26%, and long-click rate 3.34%.Source: §4.3.3 · p. 20
  • After video aggregation launched, goals per capita rose 0.63%, average QV per capita 0.55%, and the proportion of satisfied QV 1.77%.Source: §4.3.3 · p. 20
  • Brief-summary examples indicate that fine-tuning helps the model incorporate task rules for rule-guided video understanding.Source: §4.3.3 · p. 20

5 Conclusion

ARC-Hunyuan-Video introduces structured, temporally precise comprehension of real-world short videos, using audiovisual synchronization and timestamp overlays. The paper reports strong benchmark performance and downstream versatility.

5 Conclusion

  • ARC-Hunyuan-Video combines an audio encoder for fine-grained audiovisual synchronization with timestamp overlays for temporal awareness, and is trained on millions of videos annotated through an automated bootstrapped pipeline.Source: §5 · p. 20
  • Brief-summary examples show relationship inference and, in one case, recognition that a person is going home from school based on carrying a schoolbag indoors.Another example describes complex character interactions in greater detail.Source: Figure 12 · p. 21

Related papers

Loading 2507.20939v1…