PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Motion Models for Assessing Generated Videos

Kelsey Allen, Carl Doersch, Guangyao Zhou, Mohammed Suhail, Danny Driess, Ignacio Rocco, Yulia Rubanova, Thomas Kipf, Mehdi S. M. Sajjadi, Kevin Murphy, Joao Carreira, Sjoerd van Steenkiste

arXiv:2505.00209v1cs.CVcs.LG

TL;DR

Generated videos can look plausible frame by frame while their motion remains poor, a gap existing video metrics do not capture well. TRAJAN auto-encodes point tracks to assess motion across video distributions, pairs, and individual clips. It is more than three times as sensitive to synthetic temporal distortions with Fréchet distance and predicts human judgments across several video-quality dimensions, though human ratings are not always consistent.

  • Problem

    Generated videos may have plausible-looking frames but poor motion, which FVD and other popular evaluation methods do not capture well.

    Source: §1 · p. 1

  • Method

    TRAJAN auto-encodes point tracks into motion features for comparing video distributions or pairs and scoring motion inconsistencies in individual videos.

    Source: §1 · p. 1

  • Results

    More than three times more sensitive to synthetic temporal distortions with Fréchet distance, TRAJAN also predicts human ratings of motion and appearance consistency and realism across videos from 10 open-source models.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The results support evaluating generated videos through motion-focused latent comparisons and per-video reconstruction scores, rather than relying only on distributional comparisons.

    Source: §6 · p. 11

    Some videos have human ratings that no automated metric captures, and raters differ in which video aspects they value, reducing agreement.

    Source: §6 · pp. 11–12

Abstract

from arXiv · show

A current limitation of video generative video models is that they generate plausible looking frames, but poor motion -- an issue that is not well captured by FVD and other popular methods for evaluating generated videos. Here we go beyond FVD by developing a metric which better measures plausible object interactions and motion. Our novel approach is based on auto-encoding point tracks and yields motion features that can be used to not only compare distributions of videos (as few as one generated and one ground truth, or as many as two datasets), but also for evaluating motion of single videos. We show that using point tracks instead of pixel reconstruction or action recognition features results in a metric which is markedly more sensitive to temporal distortions in synthetic data, and can predict human evaluations of temporal consistency and realism in generated videos obtained from open-source models better than a wide range of alternatives. We also show that by using a point track representation, we can spatiotemporally localize generative video inconsistencies, providing extra interpretability of generated video errors relative to prior work. An overview of the results and link to the code can be found on the project page: http://trajan-paper.github.io.

1. Introduction

Generated videos can have plausible frames but incoherent motion, while existing metrics often emphasize frame content or require distribution-level comparisons. TRAJAN uses point tracks to assess motion across video distributions, pairs, and individual clips.

1. Introduction

  • Human judgments are the current quality-assessment gold standard, but their cost makes them difficult to scale for regularly tracking model improvements..Source: §1 · p. 1
  • Existing video metrics can emphasize frame-level content and typically compare distributions, leaving pairwise and individual-video evaluation unclear.Some also require access to the underlying training distribution.Source: §1 · p. 1
  • TRAJAN encodes point tracks into motion features that support distribution comparisons, video-pair comparisons, and per-video motion-inconsistency scores.Tracks are estimated with BootsTAPIR and auto-encoded by TRAJAN.Source: §1 · p. 1
  • Point tracking focuses on motion-related features rather than reconstructing the full scene, so it can assess motion separately from appearance.Figure 1 contrasts plausible motion with implausible appearance against plausible appearance with fingers morphing between frames.Source: §1 · p. 1, Figure 1 · p. 2
  • TRAJAN is more sensitive to synthetic temporal distortions than several alternatives, and its per-video reconstruction score predicts human ratings across videos from 10 open-source models.The ratings cover realism, appearance, and motion consistency.Source: §1 · p. 2
  • A human study gathered ratings of interaction quality, object and behavior realism, appearance-motion consistency, and object and camera speed; TRAJAN typically predicted them better than prior methods, with room to improve..Source: §1 · p. 2

2. Related Work

Prior video metrics address distribution comparison, paired similarity, and individual-video assessment, but many emphasize appearance or prompt semantics rather than temporal motion. TRAJAN uses learned point-track features to measure motion consistency across these settings.

Metrics for assessing video quality

  • TRAJAN is proposed as a unifying model for motion consistency across distribution comparisons, generated-to-reference pairs, and individual videos..Source: §2 · p. 2

Metrics for video distributions

  • FVD compares real and generated video distributions, but its I3D features may be biased toward frame content; alternatives include VideoMAE features and other appearance-based representations..Source: §2 · p. 2

Metrics for individual videos

  • Unlike Liu et al.'s point-track motion histograms, TRAJAN learns point-track features and is reported to perform markedly better while also evaluating individual videos..Source: §2 · p. 2

Metrics for paired videos

  • Pixel and perceptual metrics such as PSNR, SSIM, and LPIPS measure frame-level similarity, but evaluate frames independently and miss temporal coherence..Source: §2 · p. 3

Metrics for individual videos

  • Many individual-video metrics focus on prompt-semantic alignment, while TRAJAN instead assesses motion and appearance consistency independently of the prompt.The paper evaluates TRAJAN on VideoPhy and EvalCrafter as well as generated-video quality.Source: §2 · p. 3

3. Methods

TRAJAN turns point tracks into latent motion representations and reconstruction scores that support distribution, pairwise, and per-video evaluation. The methods compare this approach with motion-based and appearance-based alternatives.

3. Methods

  • Distribution comparisons use Fréchet distance between dataset latents, video pairs use latent L2 distance, and single-video quality uses an ordinal score such as reconstruction error..Source: §3 · pp. 3–4

3.1. TRAJAN

  • TRAJAN addresses orderless, occlusion-affected point tracks by encoding support trajectories and reconstructing randomly queried tracks to produce a latent and reconstruction score.The trajectory inputs include positions and occlusion flags; the decoder predicts tracks and occlusion for query points.Source: §3.1 · p. 4, §3 · p. 4
  • The encoder uses attention with occlusion masking and a fixed set of latent tokens; held-out track reconstruction encourages a dense motion representation independent of support-query selection.The resulting representation has fixed size.Source: §3.1 · p. 4
  • Average Jaccard evaluates reconstruction accuracy by combining occlusion and position accuracy across several thresholds..Source: §3.1 · p. 5

3.2.1. Motion-based

  • Motion-based baselines include point-track velocity and acceleration histograms and optical-flow warping error, which compares a flow-warped frame with the next frame while accounting for occlusion.The paper notes flow is usually estimated between consecutive frames, which may affect its ability to capture long-term patterns.Source: §3.2.1 · p. 5

3.2.2. Appearance-based

  • Appearance-based comparisons include RGB features from I3D and VideoMAE, plus MooG prediction error; the methods use feature distances or reconstruction and prediction metrics.VideoMAE-v2 is trained to fill in missing patch tubes rather than predict the future.Source: §3.2.2 · pp. 5–6

4. Human evaluation

The human evaluation uses fine-grained ratings of generated videos across four motion-related dimensions. The dimensions are intended to capture common generative-video failure modes, with real-valued judgments reflecting degrees of quality.

Human evaluation

  • The study proposes sourcing fine-grained human annotations for generated videos.Source: §4 · p. 7
  • Human annotations assess consistency, interactions, realism, and speed in generated videos.Consistency concerns appearance and motion over time; interactions concern whether object interactions are realistic; realism concerns objects and backgrounds; speed rates motion as slow, normal, or fast.Source: §4 · p. 7
  • The four rating dimensions are intended to capture failures such as unnatural shape changes, jumps or jitter, poor interactions, and implausible scenes.The examples include objects blending when they come together.Source: §4 · p. 7
  • Annotators can provide real-valued judgments to represent how strongly a video meets the evaluation criteria.Source: §4 · p. 7

5. Results

TRAJAN is evaluated across distribution comparisons, generated-to-real video pairs, individual-video assessments, and spatiotemporal error localization. Across these settings, it is consistently more sensitive to motion artifacts than alternatives, while its reconstruction errors can help locate inconsistencies.

5. Results

  • Across distributional, pairwise, and per-video evaluations, TRAJAN consistently performs better than alternatives in sensitivity to motion artifacts.The experiments cover distribution-level comparisons, generated-to-real comparisons, and per-video quality assessments.Source: §5 · p. 7

5.1. Comparing video distributions

  • TRAJAN is most sensitive to temporal distortions in distribution comparisons, while per-video TRAJAN and RAFT warp error are also particularly sensitive.Motion histograms and appearance-based methods are not sensitive in the per-video comparison; distributional comparisons have the highest overall sensitivity.Source: §5.1 · p. 7

5.2. Comparing real and generated video pairs

  • TRAJAN distances reflect motion differences more than appearance, unlike VideoMAE and I3D distances, which are more sensitive to overall recognizability.Latent distances are also not well correlated with PSNR or SSIM for any model, with the correlation lowest for TRAJAN.Source: §5.2 · pp. 8–9

5.3. Assessing generated videos individually

  • Across both datasets and all question types, TRAJAN best correlates with human judgments and performs similarly to RAFT on consistency categories.Its correlations are strongest on high-motion videos, and it also best tracks human preferences across generative models.Source: §5.3 · p. 10, Figure 7 · p. 10
  • Point-track lengths and optical-flow magnitude predict object speeds, while point-track radii predict camera speed.Source: §5.3 · p. 10
  • TRAJAN can still score unrealistic motion highly when it is smooth and easily tracked, as with a glass collapsing into a table.All metrics score that video positively despite human ratings of poor consistency and realism.Source: Figure 6 · p. 8, §5.3 · p. 10
  • After controlling for overall motion, TRAJAN strongly outperforms other metrics in explaining temporal and visual consistency and subjective likeness.For original VideoPhy physical-consistency labels, RAFT, MooG, and TRAJAN are top-performing; TRAJAN does so without fine-tuning on human data.Source: §5.3 · pp. 10–11

5.4. Spatiotemporal error localization with TRAJAN

  • Per-point Average Jaccard over short time windows localizes when and where motion and appearance inconsistencies occur.In an example, low scores identify frames where a glove morphs, with spatial errors concentrated on the glove.Source: §5.4 · p. 11

6. Conclusion

The conclusion presents TRAJAN as a point-track autoencoder for assessing generated-video motion across distributions, video pairs, and individual videos. Its results support motion-focused evaluation while identifying gaps in automated physical reasoning and consistency in human ratings.

6. Conclusion

  • TRAJAN is over three times more sensitive to synthetic temporal distortions with Fréchet distance, compares video-pair motion, and has reconstruction errors that correlate with human judgments.The reported judgments include motion consistency, appearance consistency, realism, and visual quality.Source: §6 · p. 11
  • The paper proposes TRAJAN, an architecture for auto-encoding dense point tracks to evaluate motion in generated videos.Its supported uses include distribution comparisons, generated/real video pairs, and individual-video evaluation.Source: §6 · p. 11
  • Some videos remain beyond automated metrics: future models must judge how motion should follow physical principles, not only how it moves.The conclusion also notes that individual people attend to different video aspects, resulting in low agreement on consistency ratings.Source: §6 · pp. 11–12
  • Average Jaccard can localize inconsistencies across time and visible point tracks; glove appearance morphing produces low reconstruction quality around the glove.Frame-averaged scores highlight temporal trouble spots, while per-track scores show where reconstruction is poor.Source: Figure 9 · p. 12
  • Human ratings are not a uniformly consistent gold standard, since raters differ in which aspects of temporal and appearance consistency they value.The authors call for better ways to elicit judgments and caution against using humans as the sole adjudicator between models.Source: §6 · p. 12

A. Additional Results

This appendix compares Maximum Mean Discrepancy (MMD) with Fréchet distance for evaluating video distributions. With MMD, all tested models become more sensitive to temporal corruptions, though negative spatial-only scores affect the motion-histogram result.

A.1. MMD with different backbones

  • MMD offers a kernel-based distribution distance without requiring a particular distributional form such as Gaussianity for Fréchet distance.The paper uses a polynomial kernel, k(a, b) := (a^T b + 1)^3.Source: §A.1 · p. 18
  • The squared MMD is estimated from samples drawn from the two distributions being compared, using a kernel function.The appendix states that this estimate can occasionally be negative.Source: §A.1 · p. 18
  • With MMD, all models are significantly more sensitive to temporal corruptions, and motion histograms show the largest sensitivity.That result is primarily driven by three cases where spatial-only corruptions produced negative MMD scores, which were replaced with 1e-6.Source: §A.1 · p. 18, Figure 10 · p. 18

A.2. Metric changes during training

Across WALT training checkpoints, distribution-level metrics perform similarly, while TRAJAN alone tracks training reasonably well among the single-video metrics. Additional examples show low TRAJAN distances despite substantial pixel errors when motion is correct.

A.2. Metric changes during training

  • Most metric changes occur in WALT’s first 100k training steps; after that, all metrics are more or less constant.At six checkpoints, distribution-level TRAJAN, VideoMAE, and I3D perform similarly, while TRAJAN alone tracks training reasonably well among single-video metrics.Source: §A.2 · p. 18, Figure 11 · p. 19

A.3. Additional WALT examples for generated vs. real videos

  • In two WALT examples, pixel errors are high but TRAJAN distances are low because the camera or golfer motion is correct.The pixel errors arise from unfilled image regions during a camera pan, and from an incorrect jacket color and an extra background object.Source: §A.3 · pp. 18–19
  • The examples illustrate that pixel-level differences need not correspond to a difference in motion, which TRAJAN compares in feature space.The cases involve camera motion and the golfer’s arm motion, respectively.Source: §A.3 · p. 19

A.4. Additional results on VideoPhy (Bansal et al., 2024b) and EvalCrafter (Liu et al., 2024c)

These additional results compare automated metrics with human ratings across EvalCrafter and VideoPhy settings. TRAJAN often predicts consistency and realism ratings well, while the examples also show that human judgments can be influenced by semantics and motion amount.

A.4. Additional results on VideoPhy (Bansal et al., 2024b) and EvalCrafter (Liu et al., 2024c)

  • On the full EvalCrafter dataset, RAFT-Mag is a very good predictor of human judgments across conditions; faster videos tend to receive poorer ratings.RAFT-Mag is the average optical-flow magnitude across a video.Source: §A.4 · p. 19, Table 5 · p. 19
  • On medium- and high-motion samples, TRAJAN better predicts human ratings in most cases, and raters are more consistent with one another.The paper says this suggests TRAJAN captures consistency ratings beyond overall motion amount.Source: Table 6 · p. 19, §A.4 · p. 20
  • WALT examples show that incorrect colors, extra objects, and incomplete scene content can raise pixel error even when TRAJAN’s motion distance is low.The cited cases retain correct camera or golfer-arm motion.Source: §A.4 · p. 20
  • Human ratings can diverge from motion consistency: people rate some temporally inconsistent videos as realistic and appear strongly affected by object semantics.Other examples show unrealistic or inconsistent ratings despite reasonable motion or unchanged appearance.Source: §A.4 · pp. 20–21, Figure 13 · p. 21
  • In the validation study, TRAJAN and RAFT correlate best with consistency and realism judgments, while TRAJAN performs clearly better for object interactions.The study used five raters per question and mapped Likert ratings to the slider scale for comparison.Source: §A.4 · p. 22, Table 7 · p. 22

B.1. Datasets

The experiments use UCF101 videos for spatial and spatiotemporal elastic corruptions, and include generated videos from VideoPhy. The UCF101 setup follows prior work in forming a reference distribution from the first 32 frames of train- and test-split videos.

B.1.1. UCF101

  • The UCF101 reference distribution combines the first 32 frames of videos from the train and test splits, with videos at 256 × 256 resolution.The dataset contains 13,320 videos recorded in the wild of people performing different actions.Source: §B.1.1 · p. 22
  • Elastic corruptions are applied per frame: reusing the same sampled corruption creates spatial effects, while resampling for each frame creates spatiotemporal effects.The five adopted levels vary affine and elastic distortion strength, from more global shape changes to local high-frequency distortions.Source: §B.1.1 · pp. 22–23

B.1.2. VideoPhy

  • VideoPhy uses its “solid-solid” split and includes generated videos from several competitive models, preprocessed to 256 × 256 resolution.The authors could not download the Luma Dream Machine videos, which became available after submission preparation.Source: §B.1.2 · p. 23

B.1.3. EvalCrafter

The EvalCrafter evaluation samples generated videos from multiple text-to-video models. The appendix also describes WALT generations evaluated at intermediate training checkpoints.

B.1.3. EvalCrafter

  • The EvalCrafter sample contains 104 generated videos randomly selected from 11 text-to-video models to preserve diversity while approaching VideoPhy’s size.The authors report obtaining 1,144 videos, preprocessed to 256 × 256 resolution.Source: §B.1.3 · p. 23
  • Figures 14 and 15 show UCF-101 videos with spatial and spatiotemporal corruptions, respectively, using five levels of elastic transformation from Ge et al. (2024).Source: Figure 14 · p. 24, Figure 15 · p. 25

B.1.4. WALT Generations

  • WALT is trained on Kinetics-600 for 495,000 iterations and evaluated on generated videos from six checkpoints, using the first 16 frames.The 214M-parameter frame-conditional model predicts three future latent frames from the initial two latent frames; generated videos are upsampled to 256 × 256.Source: §B.1.4 · p. 26

B.2.1. TRAJAN

TRAJAN encodes point tracks into a fixed-size motion representation and reconstructs tracks from queries, while handling occlusions and varying sampled points. The appendix also describes its training, motion baselines, track-radius measures, and human-rating study.

B.2.1. TRAJAN

  • TRAJAN encodes unordered point tracks into a fixed-size representation, masking occluded points and decoding tracks at queried space-time points to model dense motion.The authors identify permutation invariance, treating occlusions as missing data, and invariance to the particular sampled points as representation goals.Source: §B.2.1 · p. 26
  • Training includes randomly shortened clips by marking later points occluded and applying the loss only before the sampled end frame.This trains the model to encode clips shorter than 150 frames.Source: §B.2.1 · p. 27
  • 55.8 average Jaccard and 85.3 average points within threshold are reported for TRAJAN on held-out real-world data from the training distribution.The authors say the result roughly meets the performance of the underlying BootsTAPIR tracker.Source: §B.2.1 · p. 27
  • Track lengths capture accumulated movement, including jitter or changes of direction, while track radii capture the maximum distance traveled in a consistent direction.Radii are the smallest enclosing-circle radii from the start to the end of each visible track.Source: §B.2.1 · p. 27, Figure 16 · p. 28

B.2.2. Motion Histograms

  • Motion histograms describe 16-frame chunks using concatenated velocity and acceleration histograms computed over tubelets and eight angle bins.The resulting vector has 9,216 dimensions and uses densely sampled BootsTAPIR point tracks.Source: §B.2.2 · p. 28

B.2.3. RAFT

  • The RAFT baseline uses an improved optical-flow model trained on a mixture of standard flow datasets.The listed datasets include Sintel, KITTI, Kubric, TartanAir, FlyingThings, and AutoFlow.Source: §B.2.3 · pp. 28–29

B.2.4. MooG

  • MooG is a recurrent next-frame prediction model, and this implementation trains for 600K steps on a mixture of datasets.The implementation places loss on the corrected state while using the next observation to compute the loss.Source: §B.2.4 · p. 29

B.3. Human Study

  • The human study asks about interaction quality, object and behavior realism, appearance and motion consistency, and object and camera speed.Questions about consistency are conditional on raters first indicating motion; interaction-realism ratings appear only when interactions are reported.Source: §B.3 · pp. 29–30, Figure 17 · p. 30

Related papers

Loading 2505.00209v1…