Source-linked AI summary
Procedural Generation of Videos to Train Deep Action Recognition Networks
César Roberto de Souza, Adrien Gaidon, Yohann Cabon, Antonio Manuel López Peña
TL;DR
Manual labeling makes large-scale video action-recognition training costly, motivating synthetic data that can complement scarce real videos. The paper builds a procedurally controlled human-action generator and PHAV dataset, then mixes synthetic and real samples through multi-task learning. Experiments report improved recognition with small real datasets and stronger performance than comparable unsupervised generative-video models.
Problem
Manual labeling of large video collections is costly and limits training data for deep human action recognition.
Method
An interpretable physics-based procedural generator creates PHAV synthetic action videos, which are combined with real videos using multi-task representation learning.
Results
Synthetic videos significantly improve action recognition with small real-world training sets and outperform other generative video models given the same number of labeled videos.
Takeaways & Limitations
Procedurally defined surrogate action categories can complement small real datasets even when they differ from the target real-world actions.
Takeaways & Limitations
The action-composition approach has not been studied for scaling to more complex actions because of their combinatorial nature.
Abstract
from arXiv · showhide
Deep learning for human action recognition in videos is making significant progress, but is slowed down by its dependency on expensive manual labeling of large video collections. In this work, we investigate the generation of synthetic training data for action recognition, as it has recently shown promising results for a variety of other computer vision tasks. We propose an interpretable parametric generative model of human action videos that relies on procedural generation and other computer graphics techniques of modern game engines. We generate a diverse, realistic, and physically plausible dataset of human action videos, called PHAV for "Procedural Human Action Videos". It contains a total of 39,982 videos, with more than 1,000 examples for each action of 35 categories. Our approach is not limited to existing motion capture sequences, and we procedurally define 14 synthetic actions. We introduce a deep multi-task representation learning architecture to mix synthetic and real videos, even if the action categories differ. Our experiments on the UCF101 and HMDB51 benchmarks suggest that combining our large set of synthetic videos with small real-world datasets can boost recognition performance, significantly outperforming fine-tuning state-of-the-art unsupervised generative models of videos.
1. Introduction
The paper addresses the cost and difficulty of manually labeling video data by investigating procedurally generated synthetic action videos for training deep recognition models. It introduces an interpretable generative model, the PHAV dataset, and a multi-task architecture for combining synthetic and real videos.
- Manual video labeling is costly, time-consuming, error-prone, privacy-sensitive, and often impractical for modalities such as optical flow or depth.
- Procedural virtual worlds can efficiently generate large amounts of controlled, algorithmically labeled data, including modalities that humans cannot label directly.
- The work investigates synthetic human action videos whose realistic appearance and motion are conditioned on specific action categories.
- The generative model uses physics, scene composition, procedural animation, and interpretable controls over scene, actor, and action variations.
- 39,982 PHAV videos cover 35 action categories, with more than 1,000 examples per category; 21 use MOCAP data and 14 are procedurally defined.
- Synthetic videos significantly improve recognition with small real-world training sets and outperform other generative video models when paired with the same number of real examples.
2. Related work
The paper positions its approach as an early use of virtual worlds and game engines for synthetic action-recognition training videos. Unlike prior approaches focused on replaying limited motion or learning generic video tensors, it procedurally creates diverse actions and combines synthetic and real data.
- Prior synthetic-data work addressed tasks such as detection, segmentation, tracking, depth estimation, pose estimation, and scene understanding rather than action-focused video generation.
- The authors describe their work as the first investigation of virtual worlds and game engines for generating synthetic training videos for action recognition.
- The proposed approach procedurally generates new action categories, interactions, and physically plausible variations instead of merely replaying MOCAP sequences.
- Compared with VGAN, the method uses structural knowledge of physics and human actions, supports controllable scenarios and camera motions, and decouples video generation from recognition architecture.
3. PHAV: Procedural Human Action Videos
PHAV generates human action videos by combining procedural scene composition, diverse motion sources, physics-based variation, and a probabilistic model. The generator supports realistic environments, camera behaviors, synthetic action categories, and physically plausible variations while acknowledging limits on scaling atomic actions to complex events.
- 3.1. Action scene composition: PHAV places a protagonist performing a predefined action in a virtual environment with optional background and supporting actors, weather, and time-of-day conditions.Scenes include urban, natural, indoor, and sports environments, with static or moving objects and pedestrians that can physically interact with humans.
- 3.2. Camera: The generator uses a physics-based Kite camera whose sampled body and spring parameters produce varied shot types, transformations, and tracking behaviors.The camera can follow the actor, follow with delay, remain stationary, or move through handheld-like, circular, or bouncing trajectories.
- 3.3. Actions: Atomic motions from MOCAP, manual design, and programming are recombined into higher-level actions, allowing categories such as “hop” or “stagger” without requiring matching source actions.The PHAV dataset contains 35 action classes, including 21 MOCAP-grounded categories and 14 procedurally defined single-actor or interaction classes.
- 3.4. Physically plausible motion variations: Ragdoll physics, muscle perturbation, weakening, action blending, and object interactions generate diverse motion while respecting joint limits and other basic physical properties.These techniques extend beyond replaying source animations by producing physically plausible variations and interactions such as collisions, crawling, and limping.
- 3.4. Physically plausible motion variations: The authors have not studied how atomic-action decomposition scales with action complexity because composing complex events has combinatorial behavior.They leave this scaling question, illustrated by person–car collision composition, for future work.
- 3.5. Interpretable parametric generative model: The probabilistic generator represents each video through variables for the human, action, length, basic motions, variations, camera, environment, time of day, and weather.Categorical variables may use uniform or manually configured probabilities, while continuous parameters can be sampled from specified distributions.
4. Cool Temporal Segment Networks
Cool-TSN combines real and synthetic videos during training through mixed-source mini-batches and separate task-specific prediction and loss layers. Its objective updates shared representations and source-specific classifiers together.
- TSN backbone: TSN processes RGB and stacked optical-flow snippets from temporal segments and combines segment predictions through a consensus function.The implementation uses K = 3 temporal segments and the parameters from the original TSN work.
- Cool-TSN architecture: Cool-TSN mixes real-world and synthetic videos within each mini-batch for joint training.Mini-batches contain 256 videos, divided into blocks of 32 across 8 GPUs.
- Cool-TSN architecture: The architecture modifies training at mini-batch generation and at the multi-task prediction and loss layers.These changes allow the model to handle real and virtual datasets with different action-category sets.
- Multi-task objective: The loss weights each source according to its relative mini-batch proportion and activates labels only within that source’s category set.The indicator function excludes labels outside the relevant dataset’s action categories.
- Multi-task objective: Every mixed mini-batch updates shared feature layers and separate prediction layers in a common descent direction.The procedure uses standard SGD with backpropagation.
5. Experiments
Experiments evaluate PHAV-assisted action recognition on UCF-101, HMDB-51, and PHAV itself, including reduced-real-data settings and comparisons with prior methods. Synthetic videos improve recognition across modalities and datasets, with especially strong benefits for smaller real-world training sets.
- Datasets and evaluation: The evaluation uses UCF-101 and HMDB-51, reporting average accuracy over their three standard data splits.UCF-101 contains 13,320 clips across 101 classes, while HMDB-51 contains 6,849 clips across 51 categories.
- Baseline and PHAV evaluation: 82.3% average accuracy on PHAV confirms that synthetic videos contain discriminative appearance and motion patterns for TSN.The PHAV network is then used to initialize fine-tuning on UCF-101 and HMDB-51.
- PHAV-assisted training: PHAV improves recognition for all modalities and datasets through either Cool-TSN or TSN-FT, with Cool-TSN especially advantageous on HMDB-51.The authors interpret PHAV as a strong prior for TSN-FT and a regularizer for Cool-TSN.
- Reduced-data experiments: 65.8 vs. 67.8 shows that replacing half of HMDB-51 with procedural videos yields performance comparable to using the full dataset.Reducing UCF-101 to one quarter still produces a Cool-TSN model that rivals state-of-the-art methods.
- State-of-the-art comparison: The approach is competitive with state-of-the-art action recognition methods, including methods trained with substantially more manual labels.It also outperforms VGAN on UCF-101 when using the same number of manually labeled target videos.
6. Conclusion
The paper introduces PHAV, a procedurally generated synthetic action-video dataset grounded in physical and graphical priors. Its surrogate action categories can complement small real-world datasets for unrelated target actions through multi-task representation learning.
- Contribution: PHAV is a large synthetic action-recognition dataset generated by a procedural video model.The model uses prior knowledge about scenes, objects, lighting, motions, and humans to generate diverse training videos.
- Practical implication: Procedurally generated videos provide a drop-in complement to small manually labeled real-world training sets.The conclusion reports quantitative evidence for this use.
- Practical implication: Surrogate procedural categories support multi-task representation learning even when target actions differ from the synthetic actions.The target actions may have only a few real-world training examples.
- Future directions: The approach combines procedural generation from computer graphics with deep learning for action recognition.The paper identifies extensions to algorithmic ground truth generation and combinations with unsupervised generative models as future directions.
Supplementary material
The supplied supplementary-material passage contains only the authors’ names and affiliations.
- Author information: The supplementary passage lists César Roberto de Souza, Adrien Gaidon, Yohann Cabon, and Antonio Manuel López.No supplementary methods, experiments, or findings are included in the supplied passage.
1. Introduction
The supplementary material expands the publication with detailed generation-model information, dataset statistics, data modalities, and additional Cool-TSN results.
- The supplement provides an extended probabilistic graphical model and in-depth details about the parametric video-generation process.It also expands the reported generation statistics and describes additional dataset modalities.
- Additional results cover Cool-TSN performance for separate network streams.
2. Generation details
The generator uses an interpretable probabilistic model to coordinate action, human, environment, camera, timing, and variation choices. Procedural rules then instantiate diverse scenes, motions, actors, and cameras, producing a large PHAV dataset.
- Parametric model: The model organizes generation around categorical variables for human model, action, motion, variation, camera, environment, day phase, and weather.These variables are controlled by parameters and conditional probability tables.
- Parametric model: P1, P2, and P3 divide the generator into world time and weather, human models, and scene-and-action preparation.The three components are represented by probabilistic graphical models in Figures 8–10.
- Distributions: The generator samples from categorical, uniform, Bernoulli, and triangular distributions to control scene, timing, weather, and variation choices.World clock time uses triangular distributions conditioned on dawn, day, or dusk.
- Scene construction: The system places protagonists and supporting actors through environment-specific waypoint graphs, then positions cameras relative to the protagonist.Camera parameters are randomly sampled within sensible Unity-based ranges.
- Action variation: Action variation perturbs or blends selected non-critical ragdoll muscles after choosing a variation mode.Muscles are selected uniformly according to the previously chosen action category.
3. Experiments
The experiments evaluate Cool-TSN with mixtures of real-world and PHAV videos, including settings that use only fractions of the real training data. Mixing the sources helps in almost all tested cases, with optical flow contributing strongly to performance.
- Mixed training: Mixing real-world and PHAV data improves Cool-TSN performance in almost all tested cases.The analysis varies the fraction of real-world training data.
- Stream analysis: The optical-flow stream is the biggest contributor to Cool-TSN performance, including when only very small fractions of real data are used.The separate-stream results are shown for temporal optical flow and spatial RGB inputs.
- Stream analysis: The stream analysis supports the plausibility of the generated motions as useful training signals.
4. Video
The supplementary video presents random PHAV samples across selected action categories and variation types, while Figure 16 compares combined and separate TSN streams across training-data amounts.
- Supplementary video: The supplementary video samples selected PHAV action categories and divides each subsample into five variation categories.Labels identify the variation used through the Figure 18 legend.
- Supplementary video: Each video sample is labeled according to its procedural variation type.
- Stream results: Figure 16 reports TSN and Cool-TSN results for different training-data amounts using combined and separate streams.
5. Conclusion
The authors show that their procedural video-generation approach is feasible despite the challenge of incorporating physics into action videos. The supplementary material further reports performance information for RGB and optical-flow channels.
- The approach is feasible based on experimental evidence from two real-world datasets.
- Generating action videos while accounting for physics remains challenging.
- The supplementary material provides additional performance information for RGB and optical-flow channels.
- The supplementary video includes sample frames illustrating the generated videos.
- A legend explains the variations shown in the supplementary video.