PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos

Xavier Thomas, Youngsun Lim, Ananya Srinivasan, Audrey Zheng, Deepti Ghadiyaram

arXiv:2512.01803v3cs.CV

TL;DR

Evaluating generated human actions remains difficult because existing approaches do not adequately capture action correctness, anatomical plausibility, and temporal coherence. The paper learns a real-action manifold from skeletal and appearance features and measures generated-video deviations from it. Its scores strongly align with human opinion across I2V and T2V settings, while current models and MLLMs struggle to judge action correctness and temporal coherence.

  • Problem

    Existing metrics and MLLM-based judges do not accurately capture the complex motion physics of human actions, including anatomical violations and temporal incoherence.

    Source: §1 · p. 2

  • Method

    The method learns a manifold of real human actions from skeletal, appearance, and temporal features, then assesses generated videos by their deviations from that representation.

    Source: §3 · p. 4, First-order temporal coherence · p. 5

  • Results

    Manifold scores strongly align with human opinion across I2V and T2V settings and multiple models, while state-of-the-art models and MLLMs struggle to evaluate action correctness and temporal coherence.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The 10-action-class study is a proof of concept, and the framework's extensibility to more actions is shown in Sec. 5.5.

    Source: §6 · p. 15

    HMR and pose features may be unreliable under occlusion.

    Source: First-order temporal coherence · p. 5

Abstract

from arXiv · show

Despite rapid advances in video generative models, robust metrics for evaluating visual and temporal correctness of complex human actions remain elusive. Critically, existing pure-vision encoders and Multimodal Large Language Models (MLLMs) are strongly appearance-biased, lack temporal understanding, and thus struggle to discern intricate motion dynamics and anatomical implausibilities in generated videos. We tackle this gap by introducing a novel evaluation metric derived from a learned latent space of real-world human actions. Our method first captures the nuances, constraints, and temporal smoothness of real-world motion by fusing appearance-agnostic human skeletal geometry features with appearance-based features. We posit that this combined feature space provides a robust representation of action plausibility. Given a generated video, our metric quantifies its action quality by measuring the distance between its underlying representations and this learned real-world action distribution. For rigorous validation, we develop a new multi-faceted benchmark specifically designed to probe temporally challenging aspects of human action fidelity. Through extensive experiments, we show that our metric achieves substantial improvement of more than 68% compared to existing state-of-the-art methods on our benchmark, performs competitively on established external benchmarks, and has a stronger correlation with human perception. Our in-depth analysis reveals critical limitations in current video generative models and establishes a new standard for advanced research in video generation.

1 Introduction

Existing metrics struggle to evaluate human-action correctness and temporal coherence in generated videos. The paper introduces a learned action manifold, a targeted benchmark, and metrics that align with human judgments.

1 Introduction

  • Evaluating generated human actions requires assessing both whether the intended action occurs and whether body movements remain temporally coherent.The task is challenging because action recognition is ill-posed even in real videos, and generated videos add the need to assess movement over time.Source: §1 · p. 2
  • The method learns a latent manifold from semantic features of human anatomy, motion physics, and appearance, then measures how generated-video embeddings deviate from it.The manifold is learned from diverse real-world videos showing varied body geometries and performance styles.Source: §1 · p. 2
  • TAG-Bench-v0 targets fine-grained action correctness and temporal coherence, while the proposed metrics assess action consistency and temporal plausibility.The benchmark evaluates intended-action capture alongside temporal smoothness and anatomical plausibility.Source: §1 · pp. 2–3
  • The authors find that state-of-the-art models and MLLMs struggle to evaluate action correctness and temporal coherence, while manifold scores strongly align with human opinion across I2V and T2V settings.They also report that some actions challenge all generative models.Source: §1 · p. 3

2 Related Work

Prior evaluation methods measure visual similarity, video-level statistics, text alignment, or physical realism, but the surveyed approaches leave human-body distortion and action-level plausibility insufficiently addressed.

2 Related Work

  • FVD captures coarse spatiotemporal alignment, while PSNR, SSIM, and LPIPS assess reference-frame similarity without measuring temporal coherence.Physics-IQ measures spatiotemporal realism using real-world references but does not focus on body distortions or action-level correctness.Source: §2 · p. 3
  • CLIPScore captures single-frame text semantics, while VideoScore and VideoPhy assess video ratings or physical laws without capturing human-body distortions or action correctness.The passage identifies these gaps as motivation for the proposed metric.Source: §2 · pp. 3–4
  • EvalCrafter and VBench2.0 evaluate multiple video dimensions, but neither explicitly assesses whether human actions are executed plausibly over time.TAG-Bench-v0 targets human-action realism and physically plausible motion.Source: §2 · p. 4

3 Approach

The approach represents human actions through combined skeletal, appearance, and temporal features, then learns a latent action manifold. It assesses generated videos by comparing their embeddings with real-action patterns and measuring embedding-trajectory smoothness.

3.1 Human-centric feature representations

3.2 Learning a manifold of real-world actions

  • The encoder processes fixed-length frame windows through separate static and motion pathways, fuses their per-frame encodings with learned attention, and uses a [CLS]-token transformer for temporal aggregation.It also extracts frame-level embeddings for finer-grained representation.Source: Encoder architecture and training · pp. 6–7
  • Supervised contrastive learning clusters window embeddings from the same action class, while hard negatives from shuffled, repeated, or reversed frames teach sensitivity to temporal incoherence.The combined objective is a weighted sum of the semantic and hard-negative losses.Source: Training objective · p. 7

3.3 Quantitative Metrics

  • Action consistency is the distance between a generated video's averaged window embeddings and the real-action class centroid; smaller distances indicate closer alignment with real action distributions.Temporal coherence measures the smoothness of frame-embedding trajectories, with the final score averaged across windows; lower scores indicate gradual, physically consistent transitions.Source: §3.3 · p. 8

4 Telltale Action Generation (TAG)-Bench

TAG-Bench-v0 evaluates generated human actions using human ratings of action consistency and temporal coherence. Its Table 1 comparison reports that the proposed metrics outperform all methods on their respective criteria.

4 Telltale Action Generation (TAG)-Bench

  • TAG-Bench-v0 contains 300 I2V videos generated from 10 UCF-101 action classes using five open- and closed-source models, with a shared initial frame for each generation.The classes were selected for a single visible person, a full-body view, and diverse whole-body dynamics.Source: §4 · pp. 8–9
  • The proposed Scons outperforms all methods for Action Consistency, and Stemp outperforms all methods for Temporal Coherence.Table 1 reports Spearman’s ρ between metric predictions and human scores; higher is better.Source: Table 1 · p. 9
  • 246 participants rated Action Consistency and Temporal Coherence on 1–10 scales, with average inter-rater correlations of 0.716 and 0.710, respectively.The shared initial frame was used to isolate motion-generation differences from scene-layout or appearance variation.Source: §4 · p. 9

5 Experiments

The experiments compare the proposed metrics with automatic and MLLM evaluators, test generalization on an external benchmark, and use the metrics to analyze generative models and design choices.

5.1 Implementation details

  • The encoder uses TokenHMR SMPL and DWPose keypoint features, appearance and frame-difference motion features, then temporal convolutions and a Transformer; it is trained on 10 UCF101 classes.Evaluation reports Spearman rank correlation with human ratings.Source: §5.1 · p. 10

5.2 Comparison to automatic metrics and MLLMs

  • Scons correlates 0.61 with Action Consistency and Stemp 0.64 with Temporal Coherence, exceeding GPT-5 by +35.6% and +68.4% in relative gains, respectively.Both correlations are statistically significant (p < 0.05); their 95% bootstrap intervals are [0.52, 0.63] and [0.59, 0.68].Source: §5.2 · pp. 11–12

5.3 Performance on external benchmarks

  • On VBench-2.0 Human Anatomy, Stemp yields the same model ranking as human raters, including for prompted actions absent from training.Scons is not evaluated there because it requires an action centroid corresponding to the prompt.Source: Figure 3 · p. 11, §5.3 · p. 12

5.4 Comparing generative models

  • Wan2.2 has the highest TAG-Bench-v0 win ratios—0.77 for Action Consistency and 0.72 for Temporal Coherence—while no model dominates every action.Runway Gen-4 scores 0.65 and 0.61, respectively; it surpasses all models on BodyWeightSquats.Source: §5.4 · pp. 12–13
  • Shotput and ThrowDiscus are challenging across all five models, with high Scons and Stemp scores, whereas less-dynamic PushUps and PullUps are easier and higher-quality videos cluster near class centroids.Source: §5.4 · p. 13

5.5 Analysis

  • Removing supervised contrastive loss (Lsupcon) drops Action Consistency from 0.61 to 0.26; adding hard-negative loss improves Action Consistency from 0.54 to 0.61 and Temporal Coherence from 0.57 to 0.64.Including Lsupcon improves Temporal Coherence from 0.38 to 0.64.Source: §5.5 · pp. 13–14
  • Masking any input feature reduces performance; removing all motion features drops Action Consistency from 0.61 to 0.46, and replacing TokenHMR ViT features with CLIP lowers Temporal Coherence from 0.64 to 0.39.CLIP-only or DINOv2-only inputs without human-centric 3D or 2D features perform worse still.Source: §5.5 · p. 14, Table 4 · p. 14, Table 3 · p. 14
  • With 23 action classes, the model correlates 0.59 with Action Consistency and 0.56 with Temporal Coherence, versus GPT-5’s 0.46 and 0.39.The extension adds 13 action classes and follows the same human-evaluation protocol.Source: §5.5 · pp. 14–15

6 Discussion and Future Work

The paper proposes a two-part framework for evaluating generated human actions and reports alignment with human judgments and generalizability across generation models.

6 Discussion and Future Work

  • The framework separates action consistency against real videos from temporal smoothness of human anatomy.The authors identify this as a gap affecting current benchmarks and metrics.Source: §6 · p. 15
  • The main study covers 10 action classes, which the authors characterize as a proof of concept; Section 5.5 extends the framework to more actions.The conclusion names longer-form videos and integrating human-physics-based features with modern MLLMs as future work.Source: §6 · p. 15
  • Future work will extend the framework to more actions and longer videos, and explore integrating human-physics-based features with modern MLLMs.Source: §6 · p. 15

A Human Evaluation User Interface (UI)

The appendix describes the human-rating interface and filtering procedures, alongside the model-configuration table used for TAG-Bench-v0.

A Human Evaluation User Interface (UI)

  • Participants rate each generated video on Action Consistency and Temporal Coherence using a 0–10 scale, and each sees 30 videos.Action Consistency asks whether motion matches the described action; Temporal Coherence concerns natural and physically realistic motion.Source: Figure 6 · p. 21

B.1 Model configurations

  • The model-configuration table lists the image-to-video models used in TAG-Bench-v0 and their native output resolutions.Source: §B.1 · p. 21, Table 5 · p. 21

B.2 Subject rejection

  • Repeated-video filtering duplicates five of each participant’s 30 videos and retains raters within the 95th percentile of repeat-score consistency, leaving 207 of 246.Source: §B.2 · p. 22
  • A second filter rejects participants with R1 > 0.05 and R2 < 0.3, or fewer than 10 rated videos; 14 more are rejected, leaving 193.R1 and R2 are based on counts of ratings beyond the population mean by a threshold determined partly by distribution kurtosis.Source: §B.2 · p. 22
  • Inter-rater filtering excludes raters whose Spearman correlation with the remaining group is below 0.55, retaining 121 for Action Consistency and 141 for Temporal Coherence.Source: §B.2 · p. 22
  • After filtering, the study computes video-level MOS for both axes and z-score-normalized versions using the overall mean and standard deviation.Source: §B.2 · p. 22

B.3 Convergence analysis of human ratings

The human-rating variability stabilizes with roughly 9–10 raters, and the appendix illustrates the TokenHMR pipeline used to extract body and appearance features.

B.3 Convergence analysis of human ratings

  • Across five selected videos per evaluation axis, MOS standard deviation typically stabilizes after about 9–10 raters.The authors interpret this convergence as evidence that aggregated ratings are stable and based on a sufficiently large rater set.Source: §B.3 · p. 23

C.1 TokenHMR feature extraction

  • Detectron2 finds each person’s bounding box, and the cropped frame is passed to TokenHMR to predict SMPL pose, global orientation, and body shape.Source: Figure 9 · p. 24
  • Intermediate features from TokenHMR’s ViT-H/16 backbone provide the visual appearance features used by the method.The figure is adapted from TokenHMR [19].Source: Figure 9 · p. 24

C.2 Training data processing

Training videos are filtered to single-person clips, and frame representations combine 3D body features, 2D keypoints, visual features, and their temporal differences. The appendix tests the learned space’s separability, temporal sensitivity, and design choices; the cited win-ratio captions state that metric rankings align with human judgment.

E.1 Action class separability

  • Unseen test videos achieve NMI 0.98 across 10 action classes, clustering around their training-set centroids.The illustrated outliers include incoherent Shotput motion and a WallPushups video resembling squats.Source: §E.1 · p. 27, Figure 11 · p. 27, Figure 13 · p. 30

E.2 Sensitivity to temporal distortions

  • Both metrics generally rise with temporal-distortion severity, while Temporal Coherence is an exception for copied frames.Copying creates identical adjacent frames and therefore low differences in frame embeddings.Source: §E.2 · p. 28, Figure 12 · p. 28

E.3 Effect of temporal window length

  • Action Consistency improves from 0.39 to 0.61 as T grows from 4 to 32, with no further gains beyond 32 despite higher cost.The study adopts T=32.Source: §E.3 · p. 29, Table 10 · p. 29

E.5 Does providing detailed prompt help generate better actions?

  • LLM-enhanced prompts win 55% of soccer-juggling comparisons, a marginal gain that does not reliably improve action or motion realism.The comparison uses videos generated by Runway Gen-4 from 30 UCF101 initial frames.Source: §E.5 · p. 31

E.6 Generated videos as hard-negatives

  • Adding CogVideoX generated videos as hard negatives changes Action Consistency from 0.61 to 0.62 and Temporal Coherence from 0.64 to 0.63.The authors interpret the marginal changes as evidence that temporally distorted real-video negatives already provide a strong training signal.Source: §E.6 · p. 31

E.7 More training data yields better correlation

  • Adding the most representative 20% of Kinetics-700 windows raises Action Consistency from 0.61 to 0.65 and Temporal Coherence from 0.64 to 0.65.Using the top 30% instead reduces both correlations from 0.65 to 0.63.Source: §E.7 · p. 31

E.8 Attention weights

E.9 Design choices

  • Attention-weighted per-frame feature fusion achieves AC=0.61 and TC=0.64, outperforming no fusion, 1D convolution, and uniform summation.The strategy uses a learnable query to weight features before Transformer-based temporal modeling.Source: §E.9 · p. 33

E.10 Sensitivity to number of samples

Performance stabilizes with 30 training samples per class, and even 10 samples yield an Action Consistency score above GPT-5’s reported score.

E.10 Sensitivity to number of samples

  • With 10 samples per class, Action Consistency is 0.55, exceeding GPT-5’s 0.45.The full filtered UCF101 training set has 42 to 137 samples per class.Source: §E.10 · p. 33, Table 11 · p. 33
  • Performance stabilizes with only 30 samples per class.The study varies the per-class sample count while keeping the rest of the setup fixed.Source: §E.10 · p. 33, Table 11 · p. 33

F Baselines

  • The appendix identifies Table 1 as the location of the reported baseline descriptions.The supplied passage does not specify those baselines.Source: F Baselines · p. 34

F.1 Feature-based Metrics

The baselines span frame-quality, similarity, dynamics, anatomy, and multimodal evaluation measures, including metrics trained on human ratings.

F.1 Feature-based Metrics

  • Feature-based baselines include no-reference image-quality scores, frame-wise similarity measures, and adjacent-frame or sampled-frame dynamics measures.Examples include BRISQUE, SSIM, CLIP Similarity, DINO Similarity, and MSE Dyn.Source: §F.1 · pp. 34–35
  • Prompt-alignment baselines compare video or frame embeddings with text describing a person performing the target action.The listed methods include CLIP Score and X-CLIP Score.Source: §F.1 · p. 35

F.2 Multi-modal Large Language Models (MLLMs) based metrics

  • Multimodal baselines include VBench anatomy and clothing measures, VideoPhy2, VideoScore, and VideoScore2.VideoScore and VideoScore2 are trained on large-scale human ratings.Source: §F.2 · p. 35

G MLLM Prompting

The MLLM evaluation uses a temporally ordered frame grid and asks models to score action consistency and temporal coherence. In-context learning yields mixed results: a controlled test trades off the two dimensions, while an expanded test reports higher Action Consistency correlations for two models but example-sensitive gains.

G MLLM Prompting

  • The prompting example samples 40 frames into a 4×10 grid in row-major temporal order, optionally showing timestamps.The paper says the layout preserves temporal progression and spatial structure.Source: Figure 17 · p. 36

G.1 Prompt used for MLLM evaluation

  • The prompt separates action consistency—matching the target action—from temporal coherence—physical plausibility of motion and body configurations.It directs evaluators to judge visible evidence rather than infer intentions beyond the frames.Source: §G.1 · p. 36
  • Both axes use five score bands from 0.00–0.09 for completely wrong or impossible to 0.90–1.00 for strong, consistent evidence.The prompt requests JSON scores, confidence, up to three diagnostic cells, and a concise rationale.Source: §G.1 · p. 37

G.2 Limited impact of in-context learning.

  • On the 300-video test, in-context learning reduces Qwen3’s Action Consistency correlation from 0.34 to 0.29 while raising Temporal Coherence from 0.28 to 0.32.The controlled JumpingJack case study also reports saturated, near-extreme scores and no consistent improvement.Source: §G.2 · pp. 37–38, Table 13 · p. 37
  • In an expanded ten-class setting, GPT-5 reaches Action Consistency correlation 0.47 and Gemini-2.5-Pro reaches 0.34, both above their respective baselines.The in-context examples include real UCF-101 videos and CogVideoX-generated videos selected using the metric.Source: §G.2 · p. 38, Table 14 · p. 38
  • In-context learning gains are highly sensitive to selected examples and vary substantially across action classes.The authors therefore describe its effects as neither robust nor consistently transferable across actions.Source: §G.2 · p. 38

Related papers

Loading 2512.01803v3…