PaperlayerView exact revision on arXiv

Source-linked AI summary

Action100M: A Large-scale Video Action Dataset

Delong Chen, Tejaswi Kasarla, Yejin Bang, Mustafa Shukor, Willy Chung, Jade Yu, Allen Bolourchi, Theo Moutakanni, Pascale Fung

arXiv:2601.10592v1cs.CV

TL;DR

Physical-action understanding lacks large, diverse, open-vocabulary video supervision for broad domains. Action100M addresses this gap with an automated, hierarchical annotation pipeline over 1.2 million instructional videos, and VL-JEPA training demonstrates consistent scaling and strong zero-shot transfer across diverse benchmarks.

  • Problem

    Physical-action understanding requires large-scale, high-quality, diverse, open-vocabulary video data, while existing datasets remain limited in these properties.

  • Method

    Action100M uses an automated pipeline to hierarchically segment instructional videos, generate multi-level captions, and produce structured action, actor, and caption annotations.

  • Results

    VL-JEPA pretrained on Action100M shows consistent scaling and strong zero-shot transfer across diverse action recognition and text-to-video retrieval benchmarks.

  • Takeaways & Limitations

    Action100M provides a practical route for scaling open-vocabulary action understanding and supports research in video understanding and world modeling.

  • Takeaways & Limitations

    Existing caption datasets may lack temporal hierarchies, limiting supervision for modeling multi-scale activity structure.

Abstract

from arXiv · show

Inferring physical actions from visual observations is a fundamental capability for advancing machine intelligence in the physical world. Achieving this requires large-scale, open-vocabulary video action datasets that span broad domains. We introduce Action100M, a large-scale dataset constructed from 1.2M Internet instructional videos (14.6 years of duration), yielding O(100 million) temporally localized segments with open-vocabulary action supervision and rich captions. Action100M is generated by a fully automated pipeline that (i) performs hierarchical temporal segmentation using V-JEPA 2 embeddings, (ii) produces multi-level frame and segment captions organized as a Tree-of-Captions, and (iii) aggregates evidence with a reasoning model (GPT-OSS-120B) under a multi-round Self-Refine procedure to output structured annotations (brief/detailed action, actor, brief/detailed caption). Training VL-JEPA on Action100M demonstrates consistent data-scaling improvements and strong zero-shot performance across diverse action recognition benchmarks, establishing Action100M as a new foundation for scalable research in video understanding and world modeling.

1 Introduction

Action100M addresses limited open-domain physical-action understanding with a large, diverse, automatically annotated video dataset. VL-JEPA trained on it shows consistent scaling and strong zero-shot transfer across varied action and retrieval benchmarks.

  • Motivation: Action100M targets the shortage of large, high-quality, diverse training data for open-vocabulary physical-action understanding.The motivation connects action recognition to embodied learning, assistive applications, and physical-world modeling.
  • Dataset and pipeline: The dataset contains O(100 million) action instances from online instructional videos, with hierarchical segments and structured action, actor, and caption fields.Its fully automated pipeline uses frontier vision-language models, Tree-of-Captions, and Self-Refine mechanisms.
  • Dataset scale: 1.2 million videos spanning 14.6 years yield 147 million segment-level annotations and 21.3 billion English words.The pipeline required approximately 1.3 million V100 GPU hours for segmentation and captioning and 0.3 million H100/H200 GPU hours for LLM aggregation.
  • Evaluation: VL-JEPA training on Action100M is evaluated across eight downstream video action benchmarks covering fine-grained actions and high-level procedures.The benchmarks include Something-Something-v2, EPIC-KITCHENS-100, EgoExo4D Keysteps, Kinetics-400, COIN, and CrossTask.
  • Results: Scaling Action100M training data consistently improves VL-JEPA zero-shot action recognition.The scaling axis is effective batch size multiplied by number of iterations.
  • Results: VL-JEPA outperforms CLIP, SigLIP2, and Perception Encoder on average retrieval recall@1 across eight text-to-video retrieval benchmarks.The comparison includes MSR-VTT, ActivityNet, DiDeMo, MSVD, YouCook2, PVDBench, Dream-1K, and VDC-1K.

2 Related Works

Prior video datasets trade off scale, domain coverage, annotation detail, or temporal structure. Action100M combines internet-scale instructional videos with open-vocabulary, hierarchical annotations produced through automated captioning and segmentation.

  • Video action datasets: Existing action datasets are constrained by manual annotation bottlenecks, limited domain coverage, or insufficient scale.Participant-recorded datasets offer dense labels in controlled settings, internet-mined datasets broaden activity coverage, and egocentric datasets remain domain-specific.
  • Video action datasets: COIN and YouCook2 provide annotated instructional videos but have limited activity coverage and manual-annotation constraints.Internet-mined datasets expand diversity but often lack fine-grained hierarchical labels, while egocentric datasets introduce new perspectives but remain domain-specific.
  • Action100M: Action100M combines over 100 million action instances with multi-level, open-vocabulary annotations from large-scale online instructional videos.Its Tree-of-Captions derives multiple annotation levels from hierarchical temporal segmentation, combining scale with action-centric detail and temporal structure.
  • Video caption datasets: Large video-caption datasets often rely on ASR transcripts or metadata, producing short, generic captions weakly aligned with physical actions.This limits their usefulness for fine-grained action representations and detailed world models.
  • Video caption datasets: Instruction-style caption datasets produce richer descriptions but typically use fewer source videos, constraining environmental and long-horizon coverage.Their captions better capture object interactions and step-by-step activities than ASR-only captions.
  • Video caption datasets: PE Video and PLM-Video-Auto improve caption quality over pure ASR pipelines but generally assign one caption per segment without explicit temporal hierarchies.This provides limited supervision for multi-scale activity structure such as steps, sub-tasks, and overarching tasks.

3 Action100M Data Pipeline

Action100M uses a fully automated, hierarchical pipeline that converts instructional videos into multi-scale captions and structured action annotations. The design combines V-JEPA 2 temporal segmentation, complementary frame and segment captioning, and LLM-based evidence aggregation.

  • Stage 1. Temporal Segmentation: The pipeline decomposes each video into a hierarchy of temporally coherent segments, capturing fine-grained motions and longer procedural steps.Hierarchical agglomerative clustering operates on V-JEPA 2 visual embeddings with local temporal connectivity and Ward linkage.
  • Stage 2. Caption Generation: Each segment receives complementary mid-frame and video-level captions organized into a Tree-of-Captions.Mid-frame captions provide spatial detail, while segment captions capture temporal dynamics and broader context.
  • Pipeline Design: The pipeline shifts heavy reasoning to a final text-only stage after converting videos into hierarchical text representations, keeping computation manageable.Caption evidence is assembled across multiple levels rather than applying heavy VLMs directly to entire videos.
  • Stage 3. LLM Aggregation: GPT-OSS-120B aggregates captions and context at every tree node to produce brief and detailed actions, actor information, and brief and detailed video captions.Inputs include the current node, child captions, root context, metadata, and ASR transcripts.
  • Stage 3. LLM Aggregation: Three iterative Self-Refine rounds are used to improve consistency and quality of the structured annotations.Nodes shorter than four seconds are discarded before structured extraction.

4 Dataset Analysis

Action100M combines nearly 1.2 million instructional videos with multi-level textual annotations spanning procedural and everyday physical activities. Its analysis shows large annotation volume, strongly skewed toward short segments, and substantial imbalance in action concepts.

  • Source Videos: 1,199,096 face-blurred videos provide approximately 14.6 years of source content, with ASR transcripts retrieved for 72% of videos.The videos originate from HowTo100M and cover instructional content curated from 12 WikiHow categories.
  • Generated Annotations: Average annotation length increases from 3.2 words for brief actions to 95.3 words for detailed captions.Brief captions average 19.2 words and detailed actions average 27.8 words.
  • Generated Annotations: 147,092,653 annotated video segments contain an estimated 21.27B words across four annotation types.The word total comprises 0.46B brief actions, 2.83B brief captions, 3.96B detailed actions, and 14.02B detailed captions.
  • Segment Statistics: 64% of segments last 0-3 seconds, while only about 2% exceed one minute.The remaining segments are distributed across 3-10 seconds and 10 seconds to one minute, with 3.23% labeled N/A.
  • Annotation Content: Action descriptions are verb-dominated, captions contain more adjectives and object descriptors, and frequent action concepts exhibit strong imbalance.The imbalance motivates semantic resampling during training, while sunburst diagrams visualize frequent action compositions.

5 Experiments

The experiments train VL-JEPA on Action100M through staged image and video pretraining, then evaluate zero-shot action recognition and text-to-video retrieval. Results show consistent scaling, strong performance on motion-focused tasks, benefits from hierarchical annotations, and gains from semantic resampling.

  • Training and Evaluation Setup: VL-JEPA is trained in three stages: single-frame image pretraining, eight-frame Action100M video pretraining, and 32-frame video training with an unfrozen encoder.Stage 3 uses gradient accumulation and a lower learning rate because of higher memory consumption.
  • Training and Evaluation Setup: The evaluation covers zero-shot action recognition and text-to-video retrieval across eight benchmarks for each task.Action recognition uses top-1 accuracy, while retrieval uses recall@1.
  • Main Results: Action100M pretraining yields higher average action recognition and video retrieval performance, with particular strength on motion-focused and step-recognition datasets.Reported strengths include Something-something-v2, EPIC-KITCHENS-100, EgoExo4D Keysteps, COIN, and CrossTask step recognition.
  • Main Results: Scaling Action100M training data produces consistent zero-shot accuracy improvements, with a significant jump from image-only Stage 1 to video-based Stage 2.The stage plots use the number of samples seen on a log-scaled x-axis.
  • Effectiveness of Action100M Pipeline: Brief action descriptions outperform direct PLM-3B pseudo-labeling, while detailed captions outperform PLM-Video-Auto captions on most benchmarks.Ego4D atomic action descriptions improve egocentric recognition on EK-100 and EgoExo4D but not other domains.
  • Effectiveness of Semantic Resampling: Semantic resampling improves average zero-shot action classification by down-sampling frequent actions and up-sampling rare ones.The validation uses a 10M-action dataset and compares tested cluster counts against training without resampling.

6 Conclusion

The conclusion presents Action100M as a large-scale, open-domain action dataset built from procedural videos and temporally localized annotations. Experiments show consistent scaling and strong zero-shot transfer, while ablations support its annotation and resampling strategies.

  • Conclusion: Action100M contains 1.2M procedural videos annotated into 147M temporally localized segments.The dataset is described as large-scale and open-domain for action-centric video understanding.
  • Conclusion: VL-JEPA pretrained on Action100M shows consistent scaling and strong zero-shot transfer across action recognition and text-to-video retrieval benchmarks.The conclusion notes particular strengths on motion-focused and step-centric datasets.
  • Conclusion: Ablations support LLM-aggregated brief actions, hierarchical evidence, and semantic resampling as useful components of the dataset pipeline.Semantic resampling is reported to improve sample efficiency by mitigating long-tail redundancy.
  • Conclusion: Action100M provides a practical route for scaling open-vocabulary action understanding and supports future work on action anticipation, world models, and long-horizon planning.The stated future applications require reasoning over multi-scale procedural structure.

A Implementation Details

The implementation pipeline processes Tree-of-Captions nodes with a reasoning model to produce structured summaries and action annotations. It uses multi-source evidence, conservative visual grounding, and Self-Refine revision to improve faithfulness and consistency.

  • Pipeline Overview: Each Tree-of-Captions node is processed independently to generate structured action annotations.The structured output includes summary and action fields with brief and detailed descriptions.
  • Summarization: Nodes shorter than four seconds are discarded before GPT-OSS-120B generates clean structured representations from multiple caption sources.The process generates both short informative captions and comprehensive segment summaries.
  • Self-Refine: Each prompt combines the current and global Tree-of-Captions with video metadata, then processes the serialized input through three Self-Refine rounds.The model is instructed to aggregate potentially erroneous local captions into an accurate description of the specific segment.
  • Annotation Guidelines: The annotation guidelines prioritize visually observable motion and appearance while rejecting unsupported information such as unobservable speech content.They also require conservative consensus-based revision when captions are inconsistent.
  • Summarization: The summarization task produces chronological segment descriptions without exact timestamps.The output focuses on the current segment rather than the entire video.
  • Action Identification: The action-identification task outputs the main actor, a brief overall action, and a detailed procedural action description.Segments without visible actors or physical actions receive N/A for the relevant fields.

B Statistics of Duplications and Semantic Resampling

Action100M contains substantial duplicate action descriptions, so the pipeline deduplicates and clusters them semantically before analyzing coverage and resampling. The resulting clusters group similar actions while covering diverse downstream datasets.

  • Duplication Statistics: 7.58 million duplicate groups account for 141.8 million duplicate action instances.The remaining action texts occur only once in the dataset.
  • Semantic Clustering: After deduplication, k-means groups action texts into k = 10^3, 10^4, or 10^5 clusters to control action-space granularity.Neighboring clusters are ranked by cosine similarity and illustrated with sampled action texts.
  • Semantic Clustering: The clustering pipeline consistently groups actions with similar semantics, as indicated by high cosine similarity between neighboring clusters.The figure examines selected anchor clusters and their closest neighbors across tested cluster counts.
  • Downstream Coverage: The k = 104 clusters overlap with COIN, CrossTask, Epic-Kitchens-100, Kinetics-400, YouCook2, and EgoExo4D in distinct but overlapping regions.The UMAP analysis uses colored downstream-dataset samples to visualize coverage and diversity.
  • Downstream Coverage: The overlap analysis indicates that Action100M clusters provide broad coverage of the action space while reducing redundancy and enhancing diversity.The analysis also reports representational alignment with multiple downstream tasks.
Loading 2601.10592v1…