SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generation

1. Introduction

• Open-source datasets for interactive virtual humans remain scarce despite the need for large-scale specialized data.Source: §1 · p. 2
• SpeakerVid-5M introduces a large-scale, high-quality dataset for audio-visual dyadic interaction with richly annotated and aligned audiovisual pairs.Source: §1 · p. 2
• The dataset combines 770K dyadic audiovisual pairs totaling 1.8K hours with 5.2M single-speaker clips totaling 8.7K hours.Source: §1 · p. 2
• Its annotations and formats support tasks including human animation, talking-head generation, multimodal dialogue modeling, listening, and multi-turn interaction.Source: §1 · p. 2, §2.1 · p. 3
• VidChatBench uses 500 out-of-distribution speaker videos to evaluate video quality, audio-visual consistency, dialogue coherence, and identity preservation.Source: §1 · p. 2

2.2. Audio-Visual Human Video Datasets.

• Prior datasets are limited by monadic focus, variable quality, restricted release, or proprietary status, leaving dyadic interactive virtual-human data underrepresented.Source: §2.1 · pp. 2–3, §2.2 · p. 3

3. Dataset Curation

• The curation design links multimodal alignment and annotation with explicit visual and audio quality controls to produce fidelity-oriented training samples.The overall construction is organized into source collection, preprocessing, annotation, and quality filtering stages.Source: §3 · p. 3, §3.3 · p. 4, §3.4 · p. 5

3.1. Source Data Collection

• 153K YouTube audiovisual videos provide diverse two-person dialogue sources, totaling 64,386 raw hours, with 93% at 1080P or higher.Sources include interviews, news, seminars, television, variety shows, debates, and educational videos, spanning 2006 to June 2025.Source: §3.1 · p. 3

3.2. Data Pre-processing

• Preprocessing produces 3–14-second speaker clips by combining scene splitting, diarization, human tracking, lip synchronization, and ArcFace-based identity correction.The pipeline retains temporal order and human IDs, selects the two primary speakers, associates audio identities with visual boxes, and corrects inconsistent IDs across clips.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Data Annotation

• Annotations combine structured captions, speech transcripts and synchronization features, speaker IDs, pose keypoints, face-and-hand clarity scores, and motion magnitude.Qwen2.5-VL describes camera motion, entities, body orientation, framing, movements, and expressions; motion is rated from 1 to 5 using multiple prompted personas.Source: §3.3 · pp. 4–5

3.4. Quality Filter

• Quality filtering removes samples with unsuitable luminance, video quality, clarity, blur, audio reliability, or language consistency.The filters include DOVER fused scores below 0.25, bottom-5% clarity scores, face-or-hand blur scores below 0.1, and ASR thresholds on confidence, no-speech probability, compression ratio, and language match.Source: §3.4 · p. 5

4.1. Dataset Statistical Comparison.

• 5.2M single-speaker clips total 8.7K hours, while 770K two-person conversational pairs total 1.8K hours.Source: §4.1 · p. 5
• The dataset provides 93% of videos at 1080P or higher, rich annotations, synchronized audiovisual pairs, and diverse body-composition labels.Source: §4.1 · p. 5, Table 1 · p. 6

4.2. SpeakerVid-5M Dialogue Branch

• The dialogue branch pairs input and response audiovisual clips, using temporally continuous real-world conversations across varied topics and 16K speaker IDs.Source: §4.2 · p. 6

4.3. SpeakerVid-5M Single Branch

• The single-speaker branch spans full-body, half-body, frontal, and side-view videos, with skeletal, textual, and ASR annotations for conditioned generation.Source: §4.3 · pp. 6–7, Figure 4 · p. 7

4.4. SpeakerVid-5M Listening Branch

• The listening branch identifies silent listener video using audio segmentation, ASR validity and confidence, and SyncNet thresholds.Source: §4.4 · p. 7

4.5. SpeakVid-5M Multi-turn Branch

• The multi-turn branch preserves clip order and uses preceding ASR or audiovisual turns to construct contextual and sequential dialogue histories.Source: §4.5 · p. 7

4.6. Data Stratification

• The data is stratified into a 571K-clip, 1368-hour high-quality SFT subset and a 7375-hour large-scale pretraining subset using quality thresholds.Source: §4.6 · pp. 7–8

Autoregressive Talking Human Generation

• The baseline jointly predicts audio and video tokens autoregressively, then uses a shallow diffusion MLP to produce detailed videos.Source: §5 · p. 8, Figure 5 · p. 8

5.1. Autoregressive generation of video and audio

• Qwen2.5-Omni encodes input modalities, while 3D-VAE and CosyVoice2 tokenize video and audio into autoregressive chunks with positional encodings.Source: §5.1 · p. 8

5.2. Visual Optimization

• Spatial transformer refinement, frame-wise audio injection, and diffusion denoising produce temporally and spatially aligned visual latents and decoded audio.Source: §5.2 · p. 8

5.3. Progressive Training Process

• Training progresses from single-speaker visual pretraining to joint audiovisual training and high-quality dyadic dialogue fine-tuning.Source: §5.3 · p. 8, Figure 6 · p. 9
• VidChatBench evaluates conditioned generation from ASR and captions separately from dyadic generation directly from an initiator’s audiovisual input.Source: Table 2 · p. 9

6.1. The VidChatBench Benchmark.

• The VidChatBench benchmark evaluates video quality, identity preservation, dialogue coherence, audio-visual consistency, emotional alignment, and speaker identity preservation.It uses 500 representative input-output pairs with unseen speaker IDs and tailored metrics.Source: §6.1 · p. 9
• VidChatBench also measures dialogue coherence by matching generated ASR against ranked candidate responses and evaluates synchronization with SyncNet.Source: §6.1 · p. 9

6.2. Qualitative and Quantitative Results

• Direct audiovisual conditioning significantly outperforms textual conditioning, while joint audiovisual generation maintains high video quality.The comparison is between ground-truth ASR and captions versus audiovisual input from an initiator.Source: §6.2 · p. 10
• The spatial transformer improves visual metrics by refining frame-level visual tokens.Source: §6.2 · p. 10

6. Qualitative and Quantitative Results

• Training noise injection mitigates autoregressive error accumulation and improves overall video generation quality.Source: §6.2 · p. 10

• SpeakerVid-5M is a large-scale, high-quality dataset for audio-visual dyadic interactive virtual human generation, paired with VidChatBench for evaluation.Source: §7 · p. 10
• The dataset is divided into a large-scale pretraining subset and a curated high-quality subset for supervised fine-tuning.Source: §7 · p. 10
• Experiments demonstrate the effectiveness of the dataset and its high-quality supervised fine-tuning data.Source: §7 · p. 10

8. Limitation and Future Work

• The baseline is constrained by limited computational resources and is intended only as a preliminary benchmark for validating SpeakerVid-5M.The authors expect more advanced architectures and larger-scale training to improve fidelity.Source: §8 · p. 10
• SpeakerVid-5M currently focuses on single-person and two-person dyadic interaction rather than a broader interaction scope.Source: §8 · p. 10
• The paper identifies future development through stronger architectures and large-scale training, while leaving the dataset’s broader potential for later work.Source: §8 · p. 10

• Videos are standardized to 8 FPS and 480×768 resolution, with a VAE using temporal stride 4 and spatial stride 8.Source: §9 · p. 11
• Each latent frame begins with 360 tokens and is refined to 1440 tokens by the spatial transformer, while each audio chunk contains 12 tokens.Source: §9 · p. 11
• The Qwen2.5-Omni Thinker remains frozen, leaving 0.8 billion trainable parameters across the other components.Source: §9 · p. 11
• Visual pretraining and joint audiovisual training run for 15 days on 128 NVIDIA L40S GPUs, followed by five days of fine-tuning on 32 NVIDIA A800 GPUs.Source: §9 · p. 11

Basic Annotation

• Basic annotations record source access, clip boundaries, subject localization, resolution, synchronization, quality, clarity, and speaker identity for each clip.These metadata enable direct retrieval of source videos and extraction of high-quality single-person audiovisual clips.Source: §10 · p. 11

Clarity Annotation

• Frame-level and clip-level Laplacian scores quantify face and hand clarity, supporting motion-fidelity conditioning and filtering of high-clarity training data.Absolute and relative scores are computed for salient regions, then aggregated into a holistic clip score.Source: §10 · p. 11

Caption Annotation

• Caption annotations encode video quality, camera dynamics, subject composition, pose, entities, speech activity, motion, actions, and facial expression for structured conditioning and evaluation.These labels provide both holistic and fine-grained descriptions of visual content and behavior.Source: §10 · p. 11

Scene Annotation

• Scene annotations segment source videos into coherent shots and cap clips at 14 seconds, enabling concatenation for long-form synthesis and multi-turn dialogue.Each processed clip receives a unique identifier with precise start and end times.Source: §10 · p. 12

Speaker Annotation

• Speaker annotations retain dyadic exchanges only when two primary speakers account for at least 80% of total speech duration.The process diarizes speaker turns, filters to Speakers A and B, and preserves their cleaned temporal segments.Source: §10 · p. 12

ASR Annotation

• ASR annotations provide transcripts, confidence, compression ratio, no-speech probability, and language after visual and audio segmentation isolate each speaker.YOLO and speaker diarization are combined to produce single-person clips containing one person and that person’s speech.Source: §10 · p. 12

Finetuning Impact

• Fine-tuning on the refined high-quality subset substantially reduces face and hand blur, improving detail and perceptual fidelity.The pretraining model shows pronounced blur that worsens during rapid movements; filtering uses face and hand clarity scores.Source: §11 · p. 12

Prompt Design

• Tables 3 and 4 provide Qwen-VL-2.5 caption prompts that guide high-quality response generation during video annotation.The supplied passage identifies the prompts’ role but does not report their contents or comparative results.Source: §12 · p. 12

Fine-grained Hand Dynamics

• Fine-grained hand dynamics remain difficult because existing talking-head datasets contain simple motions and textual descriptions lack sufficient supervisory detail.SpeakerVid-5M adds hand keypoints, bounding boxes, and hand-blur scores for control and evaluation.Source: §13 · pp. 12–14

Annotation and Visual Examples

• SpeakerVid-5M’s prompt-based annotation framework is documented in Table 3, while visual examples cover multi-turn dialogue and diverse dyadic pairs.The supplied passages identify these visual and prompt resources but do not report additional failure-case measurements.Source: Figure 8 · p. 14, Figure 9 · p. 15, Table 3 · p. 16

Occluded or Out-of-Frame Body Parts

• Occluded or out-of-frame body-part synthesis requires anatomical and motion priors, while front-facing cropped data limits viewpoint and full-body generalization.The dataset includes full-body, upper-body, and profile views to provide more varied body configurations.Source: §13 · p. 14

Temporal Coherence in Long-Form Video Generation

• Long-form generation is constrained by scarce high-quality duration and autoregressive error accumulation, so 5–14-second clips support concatenation into 20-second-to-3-minute sequences.The described noise injection strategy is intended to improve temporal stability over extended generation horizons.Source: §13 · pp. 14–16