Source-linked AI summary
VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, Shelly Sheynin
TL;DR
Generative video models struggle with real-world motion, dynamics, and physics because pixel reconstruction can favor appearance over motion coherence. VideoJAM learns a joint appearance-motion representation during training and uses Inner-Guidance at inference, achieving state-of-the-art motion coherence while also improving perceived visual quality. It requires only two linear layers and no additional training data or model scaling, but remains limited by low resolution, RGB motion representations, and absent explicit physics encoding.
Problem
Generative video models often fail to accurately portray motion, physics, and dynamic interactions despite improvements in visual quality.
Method
VideoJAM jointly represents appearance and motion, predicts both during training, and uses the model’s evolving motion prediction for Inner-Guidance during inference.
Results
VideoJAM significantly enhances motion coherence across model sizes and motion types, surpassing proprietary state-of-the-art models while also improving perceived visual quality.
Takeaways & Limitations
Appearance and motion can be complementary when integrated in video generation, improving both visual quality and motion coherence.
Takeaways & Limitations
Limited training resolution and RGB motion representations hinder motion capture in zoomed-out scenarios, while the representation lacks explicit physics encoding for complex object interactions.
Abstract
from arXiv · showhide
Despite tremendous recent progress, generative video models still struggle to capture real-world motion, dynamics, and physics. We show that this limitation arises from the conventional pixel reconstruction objective, which biases models toward appearance fidelity at the expense of motion coherence. To address this, we introduce VideoJAM, a novel framework that instills an effective motion prior to video generators, by encouraging the model to learn a joint appearance-motion representation. VideoJAM is composed of two complementary units. During training, we extend the objective to predict both the generated pixels and their corresponding motion from a single learned representation. During inference, we introduce Inner-Guidance, a mechanism that steers the generation toward coherent motion by leveraging the model's own evolving motion prediction as a dynamic guidance signal. Notably, our framework can be applied to any video model with minimal adaptations, requiring no modifications to the training data or scaling of the model. VideoJAM achieves state-of-the-art performance in motion coherence, surpassing highly competitive proprietary models while also enhancing the perceived visual quality of the generations. These findings emphasize that appearance and motion can be complementary and, when effectively integrated, enhance both the visual quality and the coherence of video generation. Project website: https://hila-chefer.github.io/videojam-paper.github.io/
1. Introduction
VideoJAM addresses persistent failures in generated motion, physics, and temporal coherence by teaching a joint appearance-motion representation and using it to guide inference. The framework improves motion coherence and perceived visual quality with minimal architectural and data changes.
- Video models often produce deformations, physically impossible object interactions, and incorrect repetitive motion, even for basic motions represented in training data.Examples include extra limbs during gymnastics, a hoop passing through a woman, and repeated stepping on the same leg while jogging.
- The pixel-based training objective is nearly invariant to temporal perturbations during generation steps critical for determining motion.
- VideoJAM teaches a joint appearance-motion representation by modifying training to predict motion alongside appearance and adding inference-time guidance.
- The method uses the model’s evolving motion prediction as a dynamic guidance signal to steer sampling toward temporally coherent generations.
- VideoJAM enhances motion coherence and perceived visual quality without modifying training data or scaling the model.
2. Related Work
Prior work addresses video control and temporal coherence through guidance, motion or physics signals, separated content-motion generation, and image-conditioned motion representations. VideoJAM differs by teaching a temporal prior rather than requiring motion as input, while targeting generic video models.
- Diffusion-based video generation evolved from cascades and inflated image models toward Transformer-based Diffusion Transformers.
- Classifier Guidance and Classifier-Free Guidance steer generation with classifier or text signals by modifying the sampling distribution.
- Inner-Guidance requires a formulation that accounts for motion depending on both model weights and other conditions, unlike independence assumptions in prior guidance methods.
- Existing approaches use motion or physics signals for guidance or editing, whereas VideoJAM aims to teach models a temporal prior rather than take motion as input.
- Other methods separate content and motion generation, while related motion-representation methods improve coherence mainly in image-to-video models conditioned on images.
3. Motivation
The motivation experiments show that pixel-based objectives underweight temporal incoherence, while adding an explicit motion objective makes the model sensitive to frame-order perturbations. VideoJAM combines joint appearance-motion processing with inference-time motion guidance.
- Pixel-wise reconstruction can favor appearance features such as color and texture because they dominate differences, while dynamics and physics contribute less.
- The motivation experiment compares losses for original and randomly frame-permuted videos across denoising timesteps using vanilla DiT and VideoJAM.
- Until step 60, vanilla DiT is nearly invariant to frame shuffling, whereas VideoJAM shows a significant loss gap for the perturbations.
- Steps t ≤60 determine coarse motion, supporting the conclusion that the training objective is less sensitive to temporal incoherence and favors appearance over motion.
- VideoJAM’s framework jointly processes video and motion signals during training and uses noisy motion prediction to guide video prediction at inference.
4. VideoJAM
VideoJAM teaches video models a joint appearance-motion representation through dual prediction during training and Inner-Guidance during inference. The framework uses optical flow as motion representation and preserves the model’s latent dimensions while steering generation toward temporally coherent outputs.
- Motion Representation: Optical flow represents motion as a dense displacement field and is converted to RGB using per-pixel angle and normalized magnitude.The normalization assigns color by motion direction and opacity by motion magnitude, with σ = 0.15.
- Joint Appearance-Motion Training: VideoJAM modifies video models to process and predict both appearance and motion through a dual input-output format.The model receives noised video and flow latents and predicts both signals.
- Joint Appearance-Motion Training: The architecture adds two linear projection layers while maintaining the original latent dimensions and learning a unified representation for both signals.The input projection combines video and motion latents, while the output projection extracts motion from the joint representation.
- Joint Appearance-Motion Training: The training objective is extended with an explicit motion term so the model predicts a joint appearance-motion distribution.All network weights are jointly fine-tuned after modifying the input and output projections.
- Inner-Guidance: Inner-Guidance uses the model’s noisy intermediate motion prediction to steer video generation toward plausible motion during inference.The sampling distribution is modified because the motion prediction depends on the prompt and model weights.
- Inner-Guidance: The guidance formulation jointly uses conditional and unconditional predictions for text and optical-flow signals, with text dropped in 30% and flow in 20% of training steps.Unless stated otherwise, experiments use w1 = 5 and w2 = 3.
5. Experiments
Experiments evaluate VideoJAM across model sizes and motion-focused benchmarks using qualitative comparisons, automatic metrics, and human judgments. VideoJAM improves motion coherence across baselines while retaining strong quality and text-alignment results, but remains limited in zoomed-out motion and complex physics interactions.
- Experimental setup: VideoJAM is evaluated against pre-trained, open-source, and proprietary video models using qualitative and quantitative experiments.The study includes DiT-4B and DiT-30B variants and benchmarks them against models including CogVideo, Sora, Kling, Runway Gen3, Mochi, and PyramidFlow.
- Benchmarks: VideoJAM-bench tests basic, complex, rotational, and physics-related motion using 128 holdout prompts selected for clear, varied motion descriptions.The benchmark was designed to address existing evaluations that emphasize unusual or nearly static scenes over meaningful real-world dynamics.
- Qualitative results: VideoJAM-30B produces diverse motions spanning running, acrobatics, gymnastics, and physics-oriented interactions, while qualitative comparisons show more coherent motion than Sora, Kling, and DiT-30B.Baselines exhibit backward or unnatural motion, static complex actions, deformations, and physics violations such as appearing or disappearing objects.
- Quantitative results: VideoJAM outperforms all baselines in motion coherence across model sizes, automatic metrics, and human evaluations.Human raters preferred VideoJAM motion over CogVideo5B despite its larger size, and over Kling, Sora, and Gen3 in the 30B comparison with 63.8%, 68.5%, and 77.3% preference, respectively.
- Human evaluation: In direct comparisons with DiT, human raters preferred VideoJAM motion in 82.0% of 4B cases and 72.7% of 30B cases, while also favoring quality and text faithfulness.Quality preferences were 77.3% and 74.2%, and text-faithfulness preferences were 71.1% and 71.9% for the 4B and 30B models, respectively.
- Limitations: VideoJAM struggles with zoomed-out motion because limited resolution and RGB motion representations make small relative motion less informative, and it lacks explicit physics encoding for complex object interactions.The reported examples include a parachute that fails to deploy and a player’s foot failing to contact a ball before its trajectory changes.
6. Conclusions
VideoJAM identifies the training objective as a key factor in video models’ preference for appearance fidelity over motion coherence and introduces a joint appearance-motion framework to address it. Using only two additional linear layers and no additional training data, it improves motion coherence and achieves state-of-the-art results against proprietary models.
- Video models struggle with temporal coherence, including for basic motions well-represented in training datasets.
- VideoJAM equips video models with an explicit motion prior through a single latent representation that jointly captures appearance and motion.
- VideoJAM uses only two additional linear layers and no additional training data.
- VideoJAM significantly improves motion coherence and achieves state-of-the-art results against powerful proprietary models.
Impact Statement
The paper frames video generation as a technology with existing misuse concerns and states that it introduces no specific risks beyond prior advances. It also discusses why standard compositional guidance assumptions do not apply when motion predictions depend on model weights and other conditions.
- Video generation can be misused, and the paper emphasizes detecting biases and mitigating malicious use cases for safe and fair deployment.
- Standard compositional guidance assumes conditioning signals are independent of one another and the model weights.
- VideoJAM violates that assumption because its noisy motion condition is a neural-network output and also depends on text conditioning.
- The IP2P formulation incorrectly omits pθ(c1, c2) as independent of θ in this setting, and the resulting theoretical gap causes significant performance degradation in ablations.
- For IP2P comparisons, the guidance scales are w1 = 3 and w2 = 5, with their labels reversed relative to Inner-Guidance because condition order matters for IP2P.
B. Motivation Experiments
Motivation experiments test whether early denoising steps determine video motion by restarting generation from videos noised to timesteps 20, 60, and 80. The experiments use representative appearance and optical-flow motion frames to examine how generation changes across starting points.
- The SDEdit experiment noises videos to timesteps 20, 60, and 80 before continuing generation.
- Optical flow estimated with RAFT provides two motion frames alongside one representative appearance frame for each video.
- The experiment observes that coarse motion and structure are determined between steps 20 and 60, while generation from step 20 changes the entire video.
- The qualitative examples include a woman twirling a hula hoop and a giraffe stepping along a tightrope.
C. Implementation Details
VideoJAM is fine-tuned on the base models using a small subset of their original training data, with optical flow extracted for each training video. Inference uses 100 denoising steps and applies motion guidance during the first half of generation, while the models operate in compressed latent space with multiple text encoders.
- VideoJAM-4B is fine-tuned for 50,000 iterations on 32 A100 GPUs with batch size 32, while VideoJAM-30B uses 35,000 iterations on 256 A100 GPUs with batch size 256.
- Both models use 256 × 256 resolution; VideoJAM-4B has latent dimension 3072 and 32 attention blocks, while VideoJAM-30B has latent dimension 6144 and 48 attention blocks.
- Inference uses 100 denoising steps with text guidance scale w1 = 5 and motion guidance scale w2 = 3.
- Motion guidance is applied only during the first 50 generation steps because these steps determine coarse video motion.
- The models use a TAE latent space with temporal compression ×8, spatial compression 8 × 8, and text conditioning from UL2, ByT5, and MetaCLIP.
- The models are fine-tuned on fewer than 3 million samples, under 3% of the original training videos, with RAFT optical flow extracted per video.
- Baseline videos are resized to 256 resolution for comparison, with all models evaluated on five-second videos.
C.1. VBench Metrics
The evaluation separates VBench metrics into motion and appearance categories, then normalizes and combines them using the VBench-recommended weights.
- Motion metrics assess the amount and coherence of movement in generated videos.
- Appearance metrics assess per-frame aesthetic and image quality, subject consistency, and background consistency.
- All scores are normalized and combined into a weighted score using the weights suggested in the VBench paper.
D. VideoJAM-bench: Automatic Metrics Breakdown and Prompts
On VideoJAM-bench, VideoJAM balances motion quantity and coherence, outperforming or improving over relevant baselines while preserving broad video quality.
- Automatic Metrics Breakdown: VideoJAM strikes the best balance between dynamic degree and motion smoothness in the 4B model comparison.Dynamic degree measures the amount of motion, while motion smoothness measures smooth movement.
- Automatic Metrics Breakdown: VideoJAM also balances motion amount and coherence for the larger DiT-30B model.The comparison highlights a trade-off: some systems generate more motion but less coherently, while others generate smoother but subtler movement.
- Automatic Metrics Breakdown: VideoJAM outperforms the DiT-30B base model across all motion metrics and nearly all appearance metrics.
- Automatic Metrics Breakdown: VideoJAM improves almost all remaining VBench metrics, indicating that improved temporal coherence does not require sacrificing video quality.
- Prompts: VideoJAM-bench contains 128 diverse prompts covering simple, complex human, rotational, and physics-based motions.Prompts were refined using an LLM to focus on specific motion types for precise evaluation of coherent movement.
E. Movie Gen Benchmark
The Movie Gen benchmark evaluates VideoJAM against selected runnable baselines using high-motion prompts and both human and automatic metrics. VideoJAM achieves the strongest overall balance between motion and coherence while improving appearance as well.
- Evaluation Setup: The benchmark uses official prompts labeled as containing “high” motion and compares against baselines with open-source code that can run automatically.
- Results: VideoJAM strikes the best balance between the amount and coherence of generated motion on the Movie Gen benchmark.
- Results: VideoJAM outperforms all baselines by a significant margin across human-based and automatic metrics, except dynamic degree.CogVideo5B records the highest dynamic degree, while VideoJAM remains strongest on the broader comparison.
- Results: VideoJAM consistently improves over DiT-30B in both appearance and motion metrics across evaluations.
- Prompts: The benchmark contains diverse prompts for walking, gymnastics, rotational motion, physics-based actions, and other movement types.The prompt set was refined using an LLM to focus on specific motion types for evaluating coherent movement.
27.“Professional cyclist training indoors on a stationary bike trainer.”
This section lists additional motion-benchmark prompts spanning exercise, locomotion, rotational motion, object motion, and aquatic or animal movement.
- The prompts include exercise and athletic activities such as push-ups, cycling, running, skiing, swimming, and water skiing.
- The set includes locomotion prompts involving walking, jogging, running, biking, and mountain biking across varied environments.
- Rotational-motion prompts cover spiral objects, roulette wheels, fidget spinners, bicycle wheels, waterwheels, and toy tops.
- Object-motion examples include falling objects, floating and tumbling coins, washing-machine motion, balloons, and spinning basketballs.
- Animal and aquatic prompts depict horses, fish, penguins, cows, jellyfish, dogs, and people swimming in pools or oceans.