Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content

1. Introduction

• Prior open-source datasets include Panda - 70M[10], MiraData [14], OpenVid [22], and VidGen [27], using varied filtering metrics and captioning approaches such as multi-modal or structured captions.The introduction positions Koala-36M against existing dataset processing pipelines while arguing that accurate, fine-grained conditioning remains a core challenge.Source: §1 · p. 2
• The introduction identifies three challenges: aligning captions with visual content across complex transitions, filtering low-quality videos, and learning from videos with heterogeneous strengths and weaknesses.Existing filtering often uses manually selected metrics and heuristic thresholds that may not suit video generation; training heterogeneous videos uniformly can introduce ambiguity.Source: §1 · p. 2
• Koala-36M combines more accurate video splitting, detailed captions, higher-quality content, and a refined pipeline intended to improve consistency between fine-grained conditions and video content.The authors report that dataset benchmarks and video-generation model performance demonstrate the dataset’s advantage, while ablations assess the pipeline.Source: §1 · p. 2
• The pipeline detects transitions with a linear classifier on probability distributions, generates structured captions averaging 200 words, and uses a learned Video Training Suitability Score for filtering.The score network is trained on human-curated datasets, models the joint distribution of sub-metrics, and outputs a single score used as the sole filtering metric; metric conditions are also added during generation-model training.Source: §1 · p. 2
• The authors validate Koala-36M and its processing pipeline by training video-generation models on different datasets and conducting ablation studies.They report that both dataset benchmarks and generation-model performance demonstrate Koala-36M’s advantage, without giving specific results in the introduction.Source: §1 · p. 2

2.1. Video datasets

• Existing datasets include human-captioned YouCook2 [44], VATEX [30], and ActivityNet, plus automatically captioned and threshold-filtered Miradata [14], VidGen-1M [27], and OpenVid-1M [22].The literature also includes datasets for action recognition, video understanding, VQA, and retrieval, but the passages identify a need for open-source, generation-focused video-text data.Source: §2.1 · pp. 2–3
• Large datasets such as YT-Temporal-180M [40], HD-VILA-100M [37], and ACAV [16] rely on speech-recognition captions, while Panda70M [10] has scale but often simplistic captions and frequent transitions that may cause semantic incoherence.The passages describe these shortcomings as potential sources of undesirable or unpredictable transitions in generated videos.Source: §2.1 · p. 3

2.2. Video data curation

• Video data-curation research remains scarce; Stable Video Diffusion [3] reviews clipping, captioning, and filtering but is not open-source, motivating a pipeline and training-suitability metric distinct from aesthetic and technical quality assessment.The proposed metric emphasizes suitability for training rather than only aesthetic or technical video quality.Source: §2.2 · p. 3

3. Koala-36M Dataset

• 36 million clips average 13.75 seconds at 720p, with captions averaging 202 words; the dataset thus combines over 10M videos with captions longer than 200 words.Source: §3 · p. 3
• Koala-36M significantly improves consistency between fine-grained conditions and video content over Panda-70M, which shares its raw datasets.The authors attribute this advantage to their data processing pipeline.Source: Figure 2 · p. 3, §3 · p. 3

4.1. Video splitting

• A Color-Struct SVM and temporal smoothing use frame color and structure changes plus a 3σ threshold to distinguish gradual transitions from fast motion.The method addresses missed gradual transitions and false detections in fast-motion scenes associated with threshold-based splitting.Source: §4.1 · p. 4

4.2. Video captioning

• The structured caption system combines subject, action, environment, visual and camera language, and world knowledge; its captions average 200 words and improve text-video consistency.The aspects are generated separately and merged into a final caption; caption quality is evaluated for accuracy and completeness.Source: §4.2 · p. 4, §4 · p. 4

4.3. Data filtering

• VTSS models the joint distribution of video-quality sub-metrics as a single filtering score, addressing inaccurate manual thresholds that can remove high-quality videos.The score is learned by a Training Suitability Assessment Network from human-evaluated examples, then filtered using one threshold.Source: §4.3 · pp. 4–5

4.3.1. New criteria and human evaluation

• Human suitability ratings assess dynamic and static quality, naturalness, and safety; eight experts label each video, with bias elimination applied to the scores.Dynamic quality includes motion area exceeding 30% of the frame and temporal stability; naturalness criteria reject effects, transitions, subtitles, logos, and disturbing content.Source: §4.3.1 · pp. 5–6

4.3.2. Training Suitability Assessment Network

• The Training Suitability Assessment Network fuses dynamic, static, and label branches using Weight Cross-Gating; a VTSS threshold of 2.5 yields Koala-36M’s 36 million video clips.The dynamic branch uses a 3D Swin Transformer and the static branch uses ConvNext; 2.5 is selected from the roughly two-Gaussian VTSS distribution.Source: §4.3.2 · p. 6

4.4. Metrics conditioning

• Metric conditioning encodes motion, aesthetic, and clarity scores into frequency embeddings added to timestep embeddings via AdaLN, enabling fine-grained control without extra diffusion-model computation.At inference, feature scores can be set to desired values; the supplied comparison reports Koala-36M-trained generation models achieve the highest total VBench score and surpass other models on quality and semantic scores.Source: §4.4 · p. 6, Table 2 · p. 7

5.1. Experiment Setting

• Models were trained from scratch on respective datasets using the same Sora-like architecture, with 2-second 256x256 videos, 80G A100 GPUs, batch size 32, and 140M samples.The evaluation used VBench prompts expanded to address the domain gap with training captions.Source: §5.1 · p. 7

5.2. Quantitative Results

• Koala-36M-trained models surpass Panda-70M-trained models on quality and semantic scores and achieve the highest total score, with gains in aesthetic quality, object class, multi-objects, human action, and color.Source: §5.2 · p. 7, Figure 9 · p. 7

5.3. Qualitative Results

• On VBench prompts, Koala-36M yields the best video quality and text-video consistency, outperforming the larger Panda-70M dataset despite using 36M data.The qualitative comparison is shown in Fig. 10.Source: §5.3 · p. 7, Figure 10 · p. 8

5.4. Ablation Experiments

• Koala-w/o TSA improves over Panda-70M, especially in temporal consistency and flickering, supporting that re-splitting better segments transitions and detailed recaptioning strengthens visual-text learning.The reported temporal-quality examples include subject and background consistency.Source: §5.4 · pp. 7–8
• VTSS-filtered Koala-36M performs better than unfiltered variants and manually thresholded Koala-37M, indicating stronger retention of high-quality data and reduced bias from low-quality data.Source: §5.4 · p. 8
• Metrics conditions significantly improve video quality, while AdaLN-based injection provides more precise, decoupled metric control than text-encoder injection.The authors attribute the gain to sub-metrics guiding implicit modeling of data importance; the injection comparison includes style transfer with motion score.Source: §5.4 · p. 8

• Koala-36M is presented as the only dataset combining over 10M videos with high-quality fine-grained captions longer than 200 words.Source: §6 · p. 8
• Its refined pipeline improves consistency between fine-grained conditions and video content through transition detection, structured captions, filtering, and fine-grained conditioning.Source: §6 · p. 8

Limitations

• Koala-36M remains insufficient for training video-generation models with over 1B parameters, and collecting and processing a larger dataset is future work.Source: §6 · p. 8

A. Effectiveness of video splitting methods

• The transition-detection evaluation uses 10,000 annotated clips, approximately half containing transitions, and records detection precision and recall.The comparison includes CSS and two open-source Pyscenedetect-based methods.Source: A. Effectiveness of video splitting… · p. 11
• CSS outperforms both Pyscenedetect variants in transition-detection precision and recall, with high recall indicating that it rarely misses transitions.The tested variants use HSL alone or HSL with edge detection.Source: Table 3 · p. 11, A. Effectiveness of video splitting… · p. 11
• At 256 resolution, CSS has comparable CPU runtime to other methods, but it becomes significantly faster as video resolution increases.Runtime is compared across resolutions; the supplied passages do not provide specific runtime values.Source: A. Effectiveness of video splitting… · p. 11, Table 4 · p. 11, Figure 11 · p. 11

B. Elimination of Deviations between True Scores and Labeled Scores

• For suitability-score annotation, 200k randomly sampled videos are each scored by eight trained experts on a 1-to-5 scale.The procedure is designed to make annotations reflect true suitability scores more closely.Source: B. Elimination of Deviations between… · p. 11
• To reduce individual preference bias, the method standardizes each expert’s scores and rescales them using the overall scores’ mean and variance.The resulting scores align more closely with the overall score distribution.Source: B. Elimination of Deviations between… · p. 11
• To reduce label fluctuations caused by experts’ differing interpretations, the method uses the mean of the eight scores for each video.The mean is used to reduce error from score variation.Source: B. Elimination of Deviations between… · p. 11

C. Ablation Study of Training Suitability Assessment Network

• The paper reports that the TSA module is superior to Dover.The supplied passage provides no quantitative comparison values.Source: C. Ablation Study of Training… · p. 12, Table 6 · p. 12
• Adding static and feature branches improves performance, and WCGB’s cross-gating integration achieves the best results.The static branch captures static information, while the feature branch leverages additional label information.Source: C. Ablation Study of Training… · pp. 11–12, Table 5 · p. 12

D. Further Analysis on Data Filtering

• Pairwise correlations among three sub-metrics show they are not independent, supporting the claim that filtering methods neglect their joint distribution.The correlations were calculated on unfiltered data; the passage does not provide their values.Source: D. Further Analysis on Data Filtering · p. 12, Table 7 · p. 12
• Incorrectly filtered data increase as more sub-metric thresholds deviate from their optimal values, indicating accumulated filtering errors.Table 8 reports the trend for increasing numbers of inaccurate thresholds; no specific values are provided in the passage.Source: D. Further Analysis on Data Filtering · p. 12, Table 8 · p. 12

E. More Quantitative Results

• The additional evaluation further pretrains the same model on HD-VG-130M and assesses it on VBench and metrics including FVD.Source: E. More Quantitative Results · p. 12
• Non-training metrics for other datasets are also presented in Table 1 of the main paper.Source: E. More Quantitative Results · p. 12
• Experiments at 512 resolution and 4-second duration are reported to demonstrate the dataset’s superiority.Source: E. More Quantitative Results · p. 12
• Koala-36M outperforms Panda-70M and HD-VG across all reported metrics in the additional evaluation.The comparison uses the same model further pretrained on HD-VG-130M; Table 9 presents the quantitative results, but no specific values are provided here.Source: Table 9 · p. 12, E. More Quantitative Results · p. 12

F. Comparison of results from different metrics conditions

• Under the same normalized metrics score, the method is described as providing more precise control and better decoupling across metrics as video style transfers with motion score.Source: Figure 13 · p. 13

G. More Qualitative Results of Text-to-video Generation

• Figure 14 presents additional qualitative text-to-video examples labeled with Koala-36M, Koala-w/o TSA, and Panda-70M.Source: Figure 14 · p. 14, G. More Qualitative Results of… · p. 14