Source-linked AI summary
Dancing Stick Figures: An Introductory Dataset for Training Video Generation Models
Jin Hyuk Cho
TL;DR
From-scratch video-generation research needs cheaper iteration, more accessible workflows, and finer-grained diagnosis than large real-video pipelines provide. Dancing Stick Figures addresses these needs with a compact procedural dataset, an end-to-end training release, and annotations supporting controlled evaluation. The result is a reproducible environment for studying domain-specific video generation, with explicit limits on generalization and diagnostic scope.
Problem
From-scratch video-generation research faces expensive iteration, inaccessible training pipelines, and blunt scores that do not identify individual failure properties.
Method
The paper releases a deterministic synthetic dataset, generation harness, reference trainers and checkpoints, rerenderable state annotations, and validated dataset-specific metrics.
Results
The release provides a compact end-to-end environment for training and evaluating 64-frame, domain-specific video-generation models through repeatable data generation, training, inspection, and measurement.
Takeaways & Limitations
Recorded source and rendering state makes controlled dataset changes and targeted failure analysis practical within the supported synthetic domain.
Takeaways & Limitations
The domain is intentionally narrow, and the diagnostics do not generalize to realistic human video or measure all motion and topology properties.
Abstract
from arXiv · showhide
Training a video-generation model from scratch is hard for reasons that precede model design. The feedback loop is long: a failure that appears only after a training run can make each attempted fix another run. The data are hard to reach: the corpora and recipes behind strong models are large, heterogeneous, and often unreleased. And scoring is blunt: open-ended generation has no single correct output, and an aggregate score does not by itself establish whether a sample succeeds or which property failed. Dancing Stick Figures is a synthetic video dataset built against these three obstacles. For iteration speed, its 64x64, 64-frame reference task is sized for practical repeated training on a single workstation GPU. For accessibility, the release is a 0.79-GB training tier of 4,020 video clips--1,340 six-second source motions, each rendered from three cameras by a deterministic dataset-generation harness--with checkpoints and a Colab workflow that reruns the reference training pipeline at reduced budget on a 16 GB Tesla T4. For scoring, every frame retains its generating state (ARDY cskel27 joint positions, camera, body parameters, and source motion) and per-pixel depth, surface normals, and part labels. These annotations support dataset-specific metrics for visible topology and part-wise motion; corruptions expose their sensitivities and blind spots.
1. Introduction
Dancing Stick Figures addresses the high cost, poor accessibility, and blunt diagnosis of from-scratch video-generation research with a compact, connected, procedural release. Its recorded generating state supports controlled rerendering and dataset-specific failure analysis.
- Motivation: Large corpora, heavy compute, and proprietary artifacts make controlled retraining impractical for diagnosing failures that emerge only after training.The paper identifies iteration cost as a barrier because each attempted diagnosis can require another expensive run.
- Motivation: Public components are rarely packaged as one documented, runnable workflow, leaving researchers to reconstruct preprocessing, training details, and expected results.This integration burden makes small, confidence-building experiments difficult to begin and modify.
- Motivation: Open-ended generation lacks a single correct output, while motion, appearance, and prompt-following failures require distinct evaluation dimensions.The paper frames diagnosis as separate from aggregate scoring because different failure modes can occur.
- Design response: The release answers these barriers with compact repeated training, an end-to-end workflow, and procedural data whose generating state supports diagnosis.These are the three corresponding design choices stated in the introduction.
- Design response: Each rendered frame links source motion, rig and body, camera, renderer, depth, normals, and part labels, enabling one-variable rerendering and diagnostic checks.The paper limits intervention claims to recorded variables and controlled changes tested in the study.
- Release: Dancing Stick Figures connects dataset regeneration, 64-frame prompt-conditioned video training, and validated dataset-specific evaluation in one release.The release includes a deterministic harness, seed-disjoint protocol, metrics, trainers, checkpoints, and a reduced-budget Colab workflow.
2. Related Work
Related work spans inexpensive temporal prediction, procedurally rendered perception data, articulated-human synthesis, motion representations, and video-generation evaluation. Dancing Stick Figures occupies a narrower role by combining text-conditioned rendering, preserved correspondences, and controlled diagnostics.
- Controlled synthetic data: Moving MNIST made temporal prediction inexpensive, while Kubric and MOVi provided programmatically rendered scenes with dense ground truth.These works establish controlled synthetic data as a route to cheaper study and annotation-rich perception tasks.
- Controlled synthetic data: SURREAL and BEDLAM scale synthetic articulated-person rendering toward perception, whereas Dancing Stick Figures targets text-conditioned videos of one stick figure.Its small controlled domain supports affordable end-to-end retraining and testing with known corruptions.
- Video training data and motion sources: HumanVid and MiraData broaden video training data, while AMASS and HumanML3D release motion representations that Dancing Stick Figures connects through rendering.The release preserves correspondences among prompts, joints, camera and body parameters, limb labels, and pixels.
- Evaluation of generated video: FVD offers a coarse distributional reference but is not an individual-video diagnostic and can be under-sensitive to temporal corruption.The paper tests this limitation directly while separating video quality into diagnostic dimensions.
3. Dataset and generation pipeline
The dataset-generation pipeline turns ARDY motions and prompts into deterministic, multi-view clips with rich per-frame state, explicit splits, and rerenderable configurations. The release is designed to support reproducible training and inspection while preserving source-motion and renderer provenance.
- Source motions: 1,430?
- Source motions: 143 prompts generate one six-second, 20-fps motion per seed 0–9, with ARDY supplying fixed 27-joint trajectories and deterministic rendering applied afterward.The motion table preserves prompt associations and supports rebuilding the rendered dataset without rerunning ARDY.
- Release system: The release maps generation, training, evaluation, and corruption validation into one system: the harness creates data, training produces checkpoints and samples, and recorded state supports metrics.This system map makes the release an end-to-end research workflow rather than a video archive alone.
- Splits: Seeds 0–7 train, seed 8 validates, and seed 9 tests; all 134 prompts appear in every split while source motions remain disjoint.Held-out evaluation therefore measures unseen motion realisations of known prompts.
- Per-frame state: 4,020 rendered clips store projected and 3D joints, visibility, camera and body parameters, RGBA, depth, normals, 27-part segmentation, and raw motion.Rendering parameters and motion-quality flags are recorded, while flagged samples remain available for explicit filtering choices.
- Configurations: The primary 64^2 tier occupies 0.79 GB, while 128^2 preserves more detail and 32^2 supports smoke tests.The motion release also supports rerendering the full visual dataset without ARDY.
- Dataset views: A representative waving motion is shown from three cameras, across pose-spaced frames, and with segmentation, depth, normals, and projected joints.Each record additionally stores prompt, source-motion identifier, seed, camera, and body parameters.
4. Reference benchmark and evaluation
The benchmark fixes a reproducible 64-frame evaluation window and combines structural, temporal, and coarse distributional measures. Controlled corruptions validate metric sensitivities while documenting scope limitations.
- Each benchmark uses the fixed first 64 frames at native 20 fps, matching the reference model’s 3.2-second output.
- Structural and temporal diagnostics: The evaluator measures limb topology, identity, color purity, foreground fraction, and video motion signals including centroid dynamics and angular jerk.
- Evaluator validation: 96.4% of alpha-gated foreground pixels are assigned by the evaluator, with 98.2% class accuracy and 91.1% macro IoU against mapped released segmentation.
- Reference interpretation: Occlusion can produce non-zero topology scores on real rendered frames, so real-rendered scores serve as references rather than error floors.
- Reference interpretation: FVD is a secondary coarse reference whose values are comparable only within the fixed implementation, window length, and sample count.
- Corruption validation: A partial limb swap raises LIE from .063 to .266 and an extra arm raises TVR from .176 to .261, while whole-limb swaps, stretching, and hand deletion are missed.
5. Diagnostic validation and reference results
Diagnostic corruptions reveal where hand-designed metrics and FVD agree or fail, while reference runs show complementary strengths across model configurations. The results favor reading multiple diagnostics together rather than ranking models by one score.
- Diagnostic validation: Freezing zeroes motion signals, shuffling and looping create excess acceleration and angular jerk, while reversal leaves time-symmetric signals unchanged.
- Metric limitations: Reversal receives weak and inconsistent FVD response and neither FVD nor the hand-designed signals certifies directionally correct motion.
- Diagnostic validation: FVD scores freezing 620.8, shuffling 520.7, reversal 120.1, and looping 418.2 against the 114.7 real–real reference.
- I3D sanity check: Same-prompt, different-seed clips have median I3D distance 24.21 versus 32.82 for matched different-prompt pairs, while reversal changes distance only to 9.34.
- Latent-space caveat: The frozen codec has 9.5% higher held-out foreground reconstruction MSE and background-cell neutralization increases foreground error 5.5×.
- Reference results: The local mixer attains the lowest Pixel TVR (.149), jerk (.146), and FVD (282.9), while the plain image-initialized model has lower LIE (.033) and CPE (.028).
- Reference results: Mini-Wan reaches FVD 182.6 but changes representation, backbone, and auxiliary construction, so its result is not a controlled architecture comparison.
- Representation choice: Pixel-space training keeps thin-limb errors in the representation scored by the evaluator and avoids adding a separately trained video-VAE stage.
6. Accessibility and reproducibility
The release packages data generation, training, evaluation, checkpoints, and reproduction scripts into an end-to-end workflow. Reproducibility is supported by deterministic rebuilding and explicit separation between reduced-budget demonstrations and reference runs.
- Workflow: The Colab workflow rebuilds a cache, trains image and video generators for 2k steps each, samples a GIF, and compares scores with real references.
- Workflow: The reference checkpoints use 30k updates per stage, so the Colab workflow reproduces the pipeline rather than their final quality.
- Dataset rebuilding: The rebuild script recomputes body and camera settings from clip IDs and reruns the renderer without ARDY, but it does not regenerate source motions.
- Compute scope: Reference runs used an RTX PRO 6000 Blackwell, with video runs peaking at 37.8, 37.7, and 44.1 GB; lower-memory configurations were not tested.
- Comparison protocol: Table 3 compares seed-disjoint reference runs using matched 30k-step exposure for Pixel DiTs, while Mini-Wan remains a separate latent-space reference.
- Metric interpretation: The evaluator reports TVR, LIE, and CPE, while FVD is only a secondary coarse reference and is not a calibrated stick-figure quality score.
- Data provenance: The release contains procedurally rendered frames and ARDY-generated source motions, with no directly captured human images or motion sequences.
7. Limitations and future work
The paper bounds its claims to a narrow synthetic domain and dataset-specific diagnostics, whose blind spots motivate future rig-space evaluation. It also cautions against treating the release as evidence about realistic human video.
- The dataset uses one raster style, one figure, orthographic cameras, and no real-video proxy, limiting transfer to realistic human video.
- Color-based metrics measure visible topology but miss bone length, joint angles, left/right semantics, biomechanics, whole-limb swaps, proportion changes, and time reversal.
- A learned rig estimator could expose failures such as unstable bone length or end-effector errors, but its scores require validation on controlled malformations.
8. Conclusion
Dancing Stick Figures packages data, generation tools, reference models, and diagnostics into a compact environment for end-to-end video-generation research. The resulting workflow supports reproducible iteration while remaining deliberately focused on text-conditioned videos of a single rigged figure.
- The release provides a compact environment for training and evaluating small, domain-specific video-generation models end to end.
- Its measurements diagnose learned behavior, output breakdowns, and whether training or architectural changes improve particular failure modes.
- The combined data, harness, reference models, and evaluator support a reproducible cycle of modifying data, training a 64-frame model, measuring failures, and revising it.