PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation

Hui Li, Mingwang Xu, Yun Zhan, Shan Mu, Jiaye Li, Kaihui Cheng, Yuxuan Chen, Tan Chen, Mao Ye, Jingdong Wang, Siyu Zhu

arXiv:2412.00115v3cs.CV

TL;DR

Human-centric video generation lacks sufficiently diverse, high-quality data with detailed human descriptions and motion conditions. OpenHumanVid supplies such data and uses it to further pretrain an extended diffusion-transformer model, improving human-video generation while preserving general video-generation performance. The authors conclude that aligning text with human appearance, motion, and facial motion is central to high-quality outputs.

  • Problem

    Existing human-centric video generation methods and datasets lack sufficient diversity, detailed human-focused captions, and motion conditions such as skeletons and speech audio.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces OpenHumanVid and further pretrains an extended diffusion-transformer model using its captions, skeleton sequences, and speech audio.

    Source: §1 · p. 2

  • Results

    A straightforward extended model trained on OpenHumanVid improves human video generation performance while preserving performance on general video-generation tasks.

    Source: §1 · p. 2, §5.3 · p. 10, §6 · p. 10

  • Takeaways & Limitations

    Text alignment with human appearance, human motion, and facial motion is essential for producing high-quality human-centric video outputs.

    Source: §5.2 · p. 10, §6 · p. 10

    Caption generation relies on existing multimodal models, which may restrict description diversity and richness, while cultural and contextual variation in human actions may remain incompletely captured.

    Source: §5.4 · p. 10

Abstract

from arXiv · show

Recent advancements in visual generation technologies have markedly increased the scale and availability of video datasets, which are crucial for training effective video generation models. However, a significant lack of high-quality, human-centric video datasets presents a challenge to progress in this field. To bridge this gap, we introduce OpenHumanVid, a large-scale and high-quality human-centric video dataset characterized by precise and detailed captions that encompass both human appearance and motion states, along with supplementary human motion conditions, including skeleton sequences and speech audio. To validate the efficacy of this dataset and the associated training strategies, we propose an extension of existing classical diffusion transformer architectures and conduct further pretraining of our models on the proposed dataset. Our findings yield two critical insights: First, the incorporation of a large-scale, high-quality dataset substantially enhances evaluation metrics for generated human videos while preserving performance in general video generation tasks. Second, the effective alignment of text with human appearance, human motion, and facial motion is essential for producing high-quality video outputs. Based on these insights and corresponding methodologies, the straightforward extended network trained on the proposed dataset demonstrates an obvious improvement in the generation of human-centric videos. Project page https://fudan-generative-vision.github.io/OpenHumanVid

1. Introduction

OpenHumanVid addresses shortcomings in human-centric video generation by providing large-scale, high-quality videos with detailed human-focused captions and motion conditions. Further pretraining with an extended diffusion-transformer model improves human-video evaluation while retaining general video-generation performance.

  • Existing human-video generation methods struggle with appearance consistency, motion alignment, fine-grained human prompts, and motion-control signals.Source: §1 · p. 1
  • OpenHumanVid combines large-scale high-resolution videos with short, long, and structured captions, skeleton data, and speech audio.The dataset is refined from 52.3 million clips totaling 70.6K hours to 13.2 million high-quality human clips.Source: §1 · p. 2, Figure 1 · p. 2
  • Its optimized text-video alignment covers human appearance, facial motion, and body motion, addressing gaps in existing general and human-centric datasets.Source: §1 · pp. 1–2
  • Further pretraining an extended diffusion-transformer model on OpenHumanVid yields notable improvements on VBench after training with the proposed data.Source: §1 · p. 2

2. Related Work

Prior video datasets support general generation or specialized human tasks, but often lack diverse human identities and motions together with aligned captions, skeletons, and speech. OpenHumanVid addresses this gap through a multi-stage pipeline that filters videos and constructs human-focused captions and motion conditions.

Dataset Comparison

  • The dataset compares general and human video resources while adding short, long, and structured human-focused captions, skeleton sequences, and filtered speech audio.Source: Table 1 · p. 3

Data Processing Pipeline

  • Its processing pipeline performs video preprocessing and quality filtering, extracts skeleton and speech conditions, and generates, votes on, reformats, and selects captions.Source: Figure 3 · p. 4

General Video Dataset

  • Existing general video datasets provide scale but insufficient human identity and motion diversity, while human-centric datasets target specialized tasks rather than general video-generation pretraining.Source: §2 · pp. 3–4

3. Dataset

OpenHumanVid is assembled from high-quality audiovisual sources through preprocessing, quality filtering, detailed captioning, and human-motion annotations. The resulting pipeline targets reliable human appearance, motion, facial-expression, and text-video alignment.

3.1. Source Data

  • The dataset combines 134,000 audiovisual works spanning 105,000 hours, with 84.5% of videos above 720P and 52.7% above 1080P.Sources include films, television series, and documentaries across 15 genres and 58 languages.Source: §3.1 · p. 4

3.2.1 Video Preprocessing

  • Preprocessing standardizes codecs, removes subtitles and borders, detects scene boundaries, and keeps segments lasting 2 to 20 seconds.The pipeline uses H.264, CRAFT, pixel-based border analysis, and SceneDetect.Source: §3.2.1 · p. 4

3.2.2 Video Quality Filter

  • Quality filtering retains videos meeting thresholds for luminance, blur, aesthetics, global and local motion, and technical quality, improving general video quality beyond Panda-70M.The filters include luminance [10, 210], blur [0.05, 1.0), aesthetic scores of at least 4.8, motion scores [0.5, 20], and DOVER scores of at least 0.09.Source: §3.2.2 · pp. 4–5, Figure 4 · p. 5

3.2.3 Human Motion Generation

  • Caption generation combines MiniCPM and CogVLM outputs with voting, LlaMA 3.1 reformatting, and BLIP2 selection into short, long, and structured captions.Structured captions separately represent human appearance, human motion, face motion, and environment; short captions are limited to approximately 20 to 30 words.Source: §3.2.3 · p. 5

3.2.4 Human Quality Filter

  • The dataset adds DWpose skeleton sequences and synchronized speech audio, while human-quality filters assess people, regions, motion velocity, and appearance, motion, and facial-expression alignment.BLIP2 scores evaluate appearance and motion alignment, while LlaMA extracts the relevant text and supports facial-motion evaluation.Source: §3.2.3 · p. 6, §3.2.4 · p. 6

4. Network

The model extends a CogVideoX-based Diffusion Transformer baseline to improve human-centric generation while preserving pretrained knowledge. LoRA adds trainable low-rank updates, reducing adaptation overhead and mitigating catastrophic forgetting.

Baseline Diffusion Transformer Network

  • The baseline uses a causal 3D VAE, sequential latent inputs, T5 text embeddings, Expert Transformer layers, Adaptive Layer Normalization, and 3D RoPE.The architecture targets temporal dynamics and long-range dependencies while compressing video with spatial and temporal factors of 8 and 4.Source: §4 · p. 7

Extension to Baseline

  • Further training aims to preserve identity appearance while aligning facial expressions and human motions with textual captions, but risks catastrophic forgetting.The risk is loss of broad knowledge acquired during pretraining when adapting to the new human-centric task.Source: §4 · p. 7
  • LoRA represents the weight update as W′ = W + ∆W with ∆W = AB^T, training smaller low-rank matrices instead of the full weight matrix.With d < n and d < m, this reduces computational and memory overhead during training.Source: §4 · pp. 7–8
  • The proposed data trains the new LoRA parameters rather than all model parameters, improving generation of high-quality human representations aligned with textual descriptions.Figure 5 presents the extended DiT-based model design.Source: §4 · p. 8

5. Experiments

The experiments evaluate implementation choices, training-data strategies, and further pretraining for human-centric video generation, including limitations and safety risks. Results show that higher sampling rates, human-focused alignment filters, and OpenHumanVid pretraining improve consistency, motion-text alignment, and overall video metrics.

5.1. Implementation

  • Evaluation combines general VBench metrics with Face Consistency, Body Consistency, Body Semantics, and Face Semantics for human-video quality and alignment.Face consistency uses InsightFace features, body consistency uses YOLOv8 and DINO features, and semantic alignment uses BLIP2 similarity scores.Source: §5.1 · p. 8

5.2. Training Data Strategy

  • 24 FPS significantly improves appearance consistency, human-motion/text alignment, and general video metrics compared with lower sampling rates.Higher FPS also reduces artifacts in complex scenes and preserves character appearance during quick movements.Source: §5.2 · p. 8, Table 3 · p. 9, Figure 8 · p. 9
  • Human appearance text-video alignment improves face and body consistency, while human motion alignment improves body consistency and body semantics.The appearance strategy reduces facial and hand degradation, whereas the motion strategy improves adherence to prompted actions.Source: Table 4 · p. 9, Figure 8 · p. 9, §5.2 · p. 10
  • Face-motion text-video alignment improves face consistency and face semantics, producing expressions that better follow prompts.The method outperforms the baseline on prompted expressions such as sorrowful expression and smile.Source: Figure 8 · p. 9, §5.2 · p. 10

5.3. Training on OpenHumanVid

  • The extended model further pretrained on OpenHumanVid obviously improves over the baseline on both general-video and human-related evaluation metrics.The comparison incorporates higher sampling rates and human appearance, motion, and face-motion filters.Source: §5.3 · p. 10, Table 5 · p. 10

5.4. Limitations and Future Works

  • Caption generation may limit description diversity and richness, while cultural and contextual variability in human actions and expressions may remain incompletely captured.The authors propose stronger captioning models and broader identities, motion scenarios, and granular prompts as future directions.Source: §5.4 · p. 10

5.5. Safety Considerations

  • OpenHumanVid raises risks involving privacy, representation, stereotypes, bias, deepfakes, and unauthorized portrayals of individuals.The authors respond with data governance protocols, ethical standards, and calls for transparency and responsible-use guidelines.Source: §5.5 · p. 10

6. Conclusion

OpenHumanVid addresses the shortage of human-centric video data with a large-scale, high-quality dataset containing varied motion conditions. Pretraining a simple extension of existing models on it improves human video generation performance and provides a resource for future research.

6. Conclusion

  • OpenHumanVid addresses the lack of human-centric video data with a large-scale, high-quality dataset containing varied motion conditions.The dataset is intended to support a wide range of human-centric tasks.Source: §6 · p. 10
  • Pretraining a simple extension of existing models on OpenHumanVid improves human video generation performance.The authors present the dataset as a resource for advancing human video generation research.Source: §6 · p. 10

Related papers

Loading 2412.00115v3…