VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models

Problem

• Video generators can produce visually strong clips yet struggle with basic and complex motion, physical interactions, and repetitive rotation, even for motions represented in training data.These failures suggest that data and scale alone may not explain temporal incoherence.Source: §1 · pp. 1–2

Motivation

• The pixel-based training objective is nearly invariant to temporal perturbations in generation steps that are critical for determining motion, helping explain the gap between pixel quality and motion modeling.Source: §1 · p. 2

Training

• VideoJAM trains a joint appearance-motion representation by predicting both signals, using two added linear layers to combine them at the input and extract motion at the output.Its objective predicts the joint appearance-motion distribution, encouraging the model to use the motion signal.Source: §1 · p. 2

Inference

• Inner-Guidance uses the model’s evolving motion prediction as a dynamic inference-time signal to steer generation toward the joint appearance-motion distribution and away from appearance-only predictions.Because motion depends on the other conditions and model weights, the method requires a new formulation rather than assumptions used by prior guidance approaches.Source: §1 · p. 2

Results

• VideoJAM establishes state-of-the-art motion modeling, surpassing competitive proprietary models while improving perceived generation quality across model sizes and motion types.The framework requires no data modifications or model scaling and is designed to adapt to any video model.Source: §1 · p. 2

Related Work

• Video diffusion progressed from cascades and inflated image models to auto-encoder and image-conditioned approaches, then Transformer backbones following DiTs.Earlier work also advanced diffusion for image generation, editing, and personalization. Representative video approaches include cascades (Ho et al., 2022b; Singer et al., 2023), temporal-layer inflation (Guo et al., 2023; BarTal et al., 2024; Wu et al., 2023), auto-encoders (Blattmann et al., 2023b; An et al., 2023; Wang et al., 2023), image conditioning (Blattmann et al., 2023a; Girdhar et al., 2024; Hong et al., 2022), and Transformer backbones (HaCohen et al., 2024).Source: §2 · pp. 2–3
• Prior control methods include classifier guidance and CFG, while compositional score estimation and Brooks et al. (2023) rely on independence assumptions that fail for model-predicted motion.Classifier Guidance uses classifier gradients to steer toward a class, and CFG uses text instead; CFG modifies the sampling distribution but does not address noisy or multiple conditions. The compositional estimate of Liu et al. (2022) assumes conditions are independent of each other and model weights, whereas motion here is predicted by the model and depends on its weights and conditions; Brooks et al. (2023) also assume independence from model weights.Source: §2 · p. 3
• Prior work addresses the gap between pixel quality and temporal coherence through motion or physics signals, content-motion separation, and motion representations for image-to-video.Some methods use motion or physics as inputs for guidance or editing, unlike VideoJAM's goal of teaching a temporal prior. Other work separates content and motion generation; image-generation work adds spatial priors such as surface-normal and depth maps alongside a skeleton. The most similar image-to-video approaches use motion representations to improve coherence but are limited to image-conditioned models.Source: §2 · p. 3

3. Motivation

• Vanilla DiT is nearly invariant to frame shuffling through t≤60, whereas VideoJAM shows a significant loss gap under permutation, indicating sensitivity to temporal incoherence.The comparison is shown in Figure 3.Source: Figure 3 · p. 3, §3 · p. 3
• The test compares loss on original and randomly frame-permuted inputs across 35,000 videos noised at random denoising steps t∈[0,99], using vanilla DiT and fine-tuned VideoJAM.Source: §3 · p. 3
• Because steps t≤60 determine coarse motion, the findings support the hypothesis that pixel-based objectives underweight temporal information and favor appearance over motion.Source: §3 · p. 3

4.2. Joint Appearance-Motion Representations

• Optical flow provides VideoJAM’s motion representation because it is generic, flexible, and can be represented as RGB without training an additional TAE.The flow encoding assigns color by motion direction and opacity by normalized magnitude; σ = 0.15 helps keep subtler movements distinguishable.Source: §4.2 · pp. 4–5
• VideoJAM modifies the model to take noisy video and optical flow as inputs, predict both, and learn one latent representation that supports both outputs.The architecture adds dual input and output projections while retaining the original DiT latent dimensions, and the training objective includes an explicit motion term.Source: §4.2 · p. 5

4.3. Inner-Guidance

• Inner-Guidance modifies the sampling distribution to steer generation toward plausible motion using the model’s own noisy motion prediction.The motion likelihood term is intended to improve coherence by maximizing the likelihood of the generated video’s motion representation; training randomly drops text in 30% of steps and optical flow in 20%.Source: §4.3 · p. 5

5.1. Qualitative Experiments

• VideoJAM-30B generates varied basic and complex motion, while Sora, Kling, and DiT-30B exhibit backward or unnatural motion, static scenes, deformations, and physics violations.Figure 5 shows varied motion and improved physics; Figure 6 compares representative VideoJAM-bench prompts, where VideoJAM consistently produces coherent motion.Source: Figure 5 · p. 6, Figure 6 · p. 6, §5.1 · p. 7

5. Experiments

• 3 million random training-set samples—less than 3%—fine-tune DiT-4B and DiT-30B, and VideoJAM-bench tests four natural-motion categories with 128 holdout prompts.Models generate 128-frame, 24-fps videos at 256 × 256 resolution, and RAFT supplies optical flow during fine-tuning. The evaluation also uses Movie Gen; VideoJAM-bench targets real-world motion underrepresented in MGen, with prompts selected and varied in length and detail by an LLM. Baselines include open-source and proprietary models as well as the base DiT models.Source: §5 · p. 7

5.2. Quantitative Experiments

• 63.8%, 68.5%, and 77.3% motion preference over Kling, Sora, and Gen3, respectively, accompany VideoJAM’s motion-coherence advantage across both model sizes and evaluations.In direct comparisons with the base models, raters preferred VideoJAM’s motion in 82.0% of DiT-4B cases and 72.7% of DiT-30B cases. VideoJAM-30B was trained at 256 resolution on 3 million samples, while the proprietary baselines used 768 or higher resolution; the resolution disparity explains why Kling and Sora surpass it in visual quality.Source: Table 1 · p. 7, Table 2 · p. 7, §5.2 · p. 8

5.4. Limitations

• VideoJAM is less effective when zoomed-out motion occupies little of the frame and for complex physics interactions because its motion representation lacks explicit physics encoding.Limited training resolution and RGB motion representation reduce the informativeness of motion in zoomed-out scenes; absent explicit physics encoding limits handling of complex object interactions.Source: Figure 7 · p. 8, §5.4 · pp. 8–9

5.3. Ablation Study

• Removing any ablated component degrades motion coherence; dropping inference-time optical-flow prediction is most harmful, and removing motion guidance hurts more than removing text guidance.The InstructPix2Pix guidance variant gives the second-lowest motion result, while human evaluators consistently prefer VideoJAM in visual quality and text alignment over the ablations.Source: §5.3 · p. 8

6. Conclusions

• Video models struggle with temporal coherence, and the paper identifies the training objective’s preference for appearance fidelity over motion coherence as a key factor.The challenge includes basic motions that are well represented in training datasets.Source: §6 · p. 9
• VideoJAM equips video models with an explicit motion prior by capturing appearance and motion jointly in a single latent representation.The framework uses two additional linear layers and requires no additional training data.Source: §6 · p. 9
• VideoJAM significantly improves motion coherence and achieves state-of-the-art results, including against powerful proprietary models.The authors describe the framework as generic and suggest extending it with real-world priors, such as complex physics.Source: §6 · p. 9

Impact Statement

• VideoJAM aims to advance motion modeling so video generators represent the world more faithfully.The impact statement identifies this as the work’s primary goal.Source: Impact Statement · p. 9
• The authors say VideoJAM introduces no specific risks beyond those already present in previous video-generation advances.They stress detecting bias and mitigating malicious uses to support safe and fair use of generative tools.Source: Impact Statement · p. 9

A. Compositional Guidance vs. Inner-Guidance

• Liu et al. (2022) describe compositional diffusion conditioning over multiple signals, assuming the conditions are mutually independent and independent of model weights.This assumption permits dropping the conditioning-probability denominator.Source: A. Compositional Guidance vs.… · p. 12
• That independence assumption fails for VideoJAM because its noisy motion condition is a model output and depends on the text conditioning.The motion condition is denoted d_t in the passage.Source: A. Compositional Guidance vs.… · p. 12
• InstructPix2Pix (Brooks et al., 2023) extends classifier-free guidance compositionally, but its formulation also omits a conditioning term that is not independent of model weights here.The passage relates the approach to Liu et al. (2022) and Ho & Salimans (2022).Source: A. Compositional Guidance vs.… · p. 12
• The authors report that this theoretical gap causes significant performance degradation relative to Inner-Guidance in their ablations.The comparison is against the IP2P formulation.Source: A. Compositional Guidance vs.… · p. 12
• For the IP2P comparison, the experiments use guidance scales w1 = 3 and w2 = 5; unlike Inner-Guidance, IP2P is order-sensitive.The condition order is reversed relative to Eq. 8 because IP2P conditions on visual input before text, whereas Inner-Guidance is order invariant.Source: A. Compositional Guidance vs.… · p. 12

B. Motivation Experiments

• SDEdit continuations show that generation from step 20 changes the video, step 60 preserves the input’s coarse motion and structure, and both motion and appearance seem determined by step 80.Appearance may still change between steps 60 and 80; optical flow was estimated with RAFT.Source: B. Motivation Experiments · p. 12, Figure 8 · p. 12

C. Implementation Details

• 32 A100 GPUs and 50,000 iterations fine-tuned VideoJAM-4B, while 256 A100 GPUs and 35,000 iterations fine-tuned VideoJAM-30B, both at 256 × 256.The models use latent dimensions of 3072 and 6144, with 32 and 48 attention blocks, respectively; both use a fixed learning rate of 5e−6 and Flow Matching.Source: C. Implementation Details · p. 14
• 100 denoising steps use text guidance w1 = 5 and motion guidance w2 = 3, with motion guidance applied only during the first 50 steps.Videos contain 128 frames at 24 frames per second, yielding 5-second generations.Source: C. Implementation Details · p. 14
• The models operate in TAE latent space with temporal compression ×8 and spatial compression 8 × 8, using a 1 × 2 × 2 Transformer patch size and UL2, ByT5, and MetaCLIP text encoders.The TAE structure follows Polyak et al. (2024).Source: C. Implementation Details · p. 14
• Under 3 million random samples—less than 3% of the original training videos—were used for fine-tuning, with RAFT optical flow computed for each training video.Both DiT models were pre-trained on a dataset of O(100 M) videos.Source: C. Implementation Details · p. 14
• Baseline videos were resized to 256 resolution for comparison, while website visualizations used an additional super-resolution model to upscale videos to 512 × 512.All experiments except website visualizations used 256 resolution due to resource limitations; baselines generated 5-second videos and were resized spatially only.Source: C. Implementation Details · p. 14

C.1. VBench Metrics

• Appearance metrics assess per-frame aesthetic and image quality plus subject and background consistency, while motion metrics assess smoothness and the amount of motion.The motion score is intended to capture meaningful, non-static motion that is coherent and plausible.Source: §C.1 · p. 14
• All scores are normalized and combined into a weighted score using weights suggested by the VBench paper; full benchmark results appear in Appendices D and E.Source: §C.1 · p. 14

• Among proprietary baselines, Runway Gen 3 has high dynamic degree but lowest smoothness, while Kling has highest smoothness and lowest dynamic degree.Fig. 5 compares VideoJAM with Sora and Kling based on human evaluation, considered more reliable (BarTal et al., 2024); the website also includes Runway Gen 3.Source: D. VideoJAM-bench: Automatic Metrics… · p. 15
• On VBench prompts, VideoJAM improves almost all generation aspects on metrics unavailable for external benchmarks, and the motion-benchmark prompt list appears in App. F.The reported improvements indicate that the framework does not sacrifice video quality for temporal coherence.Source: D. VideoJAM-bench: Automatic Metrics… · p. 15
• On 4B, VideoJAM balances abundant generated motion with strong coherence, unlike CogVideo2B’s incoherent motion or DiT-4B’s subtle movements.Source: D. VideoJAM-bench: Automatic Metrics… · pp. 14–15, Table 4 · p. 15
• VideoJAM strikes the best balance between motion amount and coherence on 30B, outperforming DiT-30B across all motion metrics and nearly all appearance metrics.Source: Table 5 · p. 15, D. VideoJAM-bench: Automatic Metrics… · p. 15

E. Movie Gen Benchmark

• The Movie Gen evaluation uses official high-motion prompts and automatically runnable open-source baselines, alongside an apples-to-apples comparison with pretrained DiT-30B.The benchmark is larger than VideoJAM-bench and contains mostly less relevant prompts, motivating the restricted baseline set.Source: E. Movie Gen Benchmark · p. 15
• VideoJAM’s 30B model consistently improves over DiT-30B on appearance and motion metrics across evaluations, and improves almost all VBench metrics on VBench prompts.The comparison to DiT-30B assesses VideoJAM’s direct impact on the pretrained model.Source: E. Movie Gen Benchmark · p. 16, Table 6 · p. 16
• VideoJAM outperforms all baselines by significant margins across human-based and automatic metrics, except CogVideo5B leads dynamic degree, while balancing motion amount and coherence.CogVideo5B produces the most motion but is least coherent; Mochi generates less motion.Source: Table 7 · p. 16, E. Movie Gen Benchmark · p. 16, Table 8 · p. 16

F. VideoJAM-bench Prompts

• VideoJAM-bench contains 128 prompts spanning simple, complex human, rotational, and physics-based motions, refined to focus on motion types and varied in detail and camera instructions.These design choices aim to enable clear evaluation of coherent movement across scenarios.Source: F. VideoJAM-bench Prompts · p. 16

13.“A white kitten playing with a ball.”

• The supplied prompt examples include a runner dodging traffic, basketball play, floating basketballs, and a dog playing with a ball.These examples are among the prompts listed under the subsection headed “13. ‘A white kitten playing with a ball.’”Source: 13.“A white kitten playing with a ball.” · p. 17

19.“A woman doing acrobatic exercises on a pole in the gym.”

• The supplied prompt examples under this subsection include a young man performing a cartwheel and a golden retriever playing fetch.Other listed prompts describe spinning balls, jogging on a forest trail, jumping, and marching.Source: 19.“A woman doing acrobatic exercises… · p. 18

30.“A man jumping in a pool.”

• Indoor and outdoor exercise scenes include push-ups, cycling, stationary-bike workouts, and a runner in a rainstorm.The captions describe both workout activity and close views of movement, including pedaling and running legs.Source: 30.“A man jumping in a pool.” · p. 18, 38.“A young girl coloring at her desk.” · p. 18

38.“A young girl coloring at her desk.”; 40.“Tracking camera shot. A kangaroo hops swiftly across an open grassy plain.”

• Running and walking captions range from a rain-soaked jog and rocky coastal path to a country road and riverside route.The descriptions emphasize feet, rhythmic steps, and the surrounding terrain or water.Source: 38.“A young girl coloring at her desk.” · p. 18, 40.“Tracking camera shot. A kangaroo… · p. 19, 47.“A puppy runs through a grassy… · p. 19

47.“A puppy runs through a grassy field.”; 50.“Sweet Cherries on Stems Colliding and Splashing Water Droplets”; 53.“A green helicopter taking off from an airport runway.”

• Other scenes show a washing-machine cycle, riverside jogging, and a hand spinning a fidget spinner.The descriptions include rotating drum motion, steady footsteps, and repeated hand-driven spinning.Source: 47.“A puppy runs through a grassy… · p. 19, 50.“Sweet Cherries on Stems Colliding… · p. 19, 53.“A green helicopter taking off from… · p. 19

55.“A windmill spinning in a green field.”; 59.“A man riding a mountain bike on a dirt trail.”; 71.“A woman doing push-up exercise on a beach at sunset.”

• Motion examples include bicycle and waterwheel rotations, a spinning toy top, and a woman running through a field.The captions describe consistent rotation and movement through forest or grassy settings.Source: 55.“A windmill spinning in a green… · p. 19, 59.“A man riding a mountain bike on a… · p. 19, 71.“A woman doing push-up exercise on… · p. 20

74.“A person engaging in a boxing workout at a gym.”; 83.“Women’s legs walk into the sea with waves.”; 91.“A woman is doing push-ups on a mat in the studio.”

• Further action descriptions feature a running horse, stair-running legs, a treadmill walker, soccer practice, and dancers performing turns.They span gym, city, and stage settings, including a four-spin pirouette.Source: 74.“A person engaging in a boxing… · p. 20, 83.“Women’s legs walk into the sea… · p. 20, 91.“A woman is doing push-ups on a mat… · p. 21

99.“A young woman enjoys a cup of coffee on a balcony.”; 102.“Aerial tracking camera shot. A white semi-truck drives on a highway.”; 106.“A baboon eats a mango.”

• Other captions describe drumming, apples rotating through splashing liquid, and a woman changing poses on an aerial hoop.The aerial-hoop description notes repeated changes in body position against a wall.Source: 99.“A young woman enjoys a cup of… · p. 21, 102.“Aerial tracking camera shot. A… · p. 21, 106.“A baboon eats a mango.” · p. 22

110.“A woman doing a pirouette in an empty dance studio.”; 112.“A brunette woman doing some acrobatic elements on aerial hoop outdoors.”; 115.“A woman dancing in a gym. The woman is spinning around repeatedly.”; 117.“Arc camera shot. A young woman doing stretches on a beach.”

• Dance and performance descriptions include hip-hop, pole movement, ducks walking in a row, and a woman spinning among sunflowers.The sunflower scene specifies a counterclockwise spin.Source: 110.“A woman doing a pirouette in an… · p. 22, 112.“A brunette woman doing some… · p. 22, 115.“A woman dancing in a gym. The… · p. 22

121.“A cheetah running in the Savannah.”; 124.“A woman doing sit-ups at a gym.”

• The final examples describe a dog leaping for a frisbee, a child riding a bicycle, and a runner moving quickly along a suburban sidewalk.The dog’s leap occurs as people in the background clap and cheer.Source: 121.“A cheetah running in the Savannah.” · p. 22, 124.“A woman doing sit-ups at a gym.” · p. 22