PaperlayerView exact revision on arXiv

Source-linked AI summary

TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining

Shih-Ying Yeh, Daniel Z. Kaplan, Xuehai Wang, Fu-En Yang, Min-Hung Chen, Shang-Hong Lai

arXiv:2609.33419v1cs.CV

TL;DR

Prior video SSL comparisons often vary multiple recipe choices at once, obscuring which architecture-objective combinations produce motion-prioritized representations. TT-VidT pairs compact temporal transfer with first-frame-anchored Diff Compression, and it leads four motion-heavy fine-tuning benchmarks simultaneously. The paper bounds that profile on appearance-sensitive evaluations and under a non-neutral comparison condition.

  • Problem

    Prior video SSL comparisons often entangle architecture, objective, data exposure, schedule, and scale, making it difficult to attribute motion-oriented behavior to a specific choice.

    Source: §1 · p. 1

  • Method

    TT-VidT combines a DINOv3-initialized per-frame spatial path with compact temporal transfer, reconstructing target frames from a first-frame anchor and frame-specific motion tokens.

    Source: §5 · p. 10

  • Results

    TT-VidT is the only final-comparison method to lead Jester, Something-Something V2, ARID, and Diving48 fine-tuning simultaneously.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The sweep indicates that the strong motion-sensitive regime comes from pairing TT3D with Diff Compression, not either component alone.

    Source: §5 · p. 10

    The comparison is not neutral for every objective because uniform no-augmentation may handicap non-TT3D objectives, and IARD and HMDB51 favor alternative configurations.

    Source: §4.1 · p. 6, §4.2 · p. 7

Abstract

from arXiv · show

Comparisons in video self-supervised learning often evaluate complete training recipes rather than isolating the method itself: architecture, objective, data exposure, schedule, scale, and decoder capacity can all vary at once. This makes it hard to identify which choices yield motion-prioritized representations, whose gains concentrate on frame-to-frame change while retaining useful appearance. We address this with a matched $4 \times 6 = 24$ architecture-objective study at roughly 170M ~ 190M encoder scale on $\sim$1.7M OpenVid and Moments-in-Time v2 clips for 8 epochs, and propose TT-VidT. TT-VidT combines a DINOv3-initialized ViT-B/16 per-frame spatial path with a compact Temporal Transfer Layer, trained by Diff Compression to reconstruct target frames from a first-frame appearance anchor and frame-specific motion tokens. The sweep shows that TT3D with Diff Compression, not either component alone, enters the strongest motion-sensitive regime, and decoder ablations favor a compact video-pretrained decoder. In final comparison, TT-VidT leads Jester, Something-Something V2, ARID, and Diving48 fine-tuning simultaneously, improving over the strongest non-TT row by 54% ~ 121%, while using 48% fewer encoder FLOPs than DisMo and 55% fewer than VideoMAE or V-JEPA2. HMDB51, IARD, and EPIC-Kitchens bound the claim.

1 Introduction

The introduction argues that recipe-level differences obscure which video self-supervised choices yield motion-prioritized representations. TT-VidT uses a matched architecture-objective comparison and a decoupled spatial-temporal design to study that question.

1 Introduction

  • Prior comparisons entangle architecture, objective, data exposure, schedule, and parameter scale, making motion-oriented behavior hard to attribute to a specific choice.Action-recognition accuracy can also rely on appearance cues such as objects, scenes, actors, clothing, and camera context [Liu et al., 2021, Li et al., 2018, Kowal et al., 2022, Fioresi et al., 2025].Source: §1 · p. 1
  • TT-VidT preserves a wide first-frame appearance anchor while routing frame-specific information through a compact temporal path trained to reconstruct target frames.The narrow frame-specific bottleneck is intended to make motion-sensitive improvements interpretable as useful temporal transfer under the matched recipe, not to remove appearance information.Source: §1 · p. 2
  • The study compares 24 architecture-objective combinations under a matched recipe at roughly 170M–190M encoder scale, using about 1.7M clips for 8 epochs.The clip mixture is OpenVid and Moments-in-Time v2; evaluation includes three canonical baselines.Source: §1 · p. 2

2 Related Work

The related-work discussion positions TT-VidT among motion-aware objectives, large-scale video pretraining, and image-pretrained models with temporal modules. It also frames evaluation as a diagnostic across benchmarks with different appearance and motion demands.

2 Related Work

  • DisMo is the closest comparison in scale and conceptual lineage, using appearance augmentation to reduce identity leakage while preserving motion conditioning [Ressler-Antal et al., 2025].The matched DisMo-style row shares TT-VidT’s DINOv3-initialized 2D path and 3D transformer setup where applicable, but uses DisMo’s motion-control objective rather than Diff Compression.Source: §2 · p. 3
  • VideoMAE and ARVideo use masked reconstruction or autoregressive prediction, but their reported data exposure exceeds the matched recipe, preventing a one-to-one controlled comparison [Tong et al., 2022, OpenMMLab, 2024, Ren et al., 2024].Motion-aware MAE variants target temporal information through altered masking, appearance-motion decoders, temporal differences, trajectories, synthetic motion, or shortcut removal [Bandara et al., 2023, Song et al., 2022, Yang et al., 2022, Sun et al., 2023, Thoker et al., 2025b, Vandeghen et al., 2026, Dave et al., 2024].Source: §2 · p. 3
  • V-JEPA 2, VideoMAE v2, Toto, VideoMAP, NExT-Vid, and SALT provide upper-bound context, but use different model sizes, data volumes, schedules, or teacher compute [Bardes et al., 2024, Assran et al., 2025, Wang et al., 2023a, Rajasegaran et al., 2025, Liu et al., 2025, Li et al., 2025a,b].The paper frames TT-VidT as a small-recipe design study rather than a direct comparison with those native large-scale regimes.Source: §2 · p. 3
  • Image-pretrained temporal-module approaches provide architectural precedent for combining an image foundation path with video-specific computation, not matched-recipe SSL baselines [Wu et al., 2025, TV et al., 2025, Li et al., 2025b, Wang et al., 2023b, Pan et al., 2022, Yang et al., 2023, Qing et al., 2023, Li et al., 2024].TT-VidT jointly trains all parameters in its DINOv3-initialized 2D path and narrow temporal pathway.Source: §2 · p. 3
  • The evaluation uses a diagnostic ladder because benchmarks vary in motion demand and appearance bias, while single-score summaries can hide domain and probe variation [Kuehne et al., 2011, Liu et al., 2021, Fioresi et al., 2025, Materzynska et al., 2019, Goyal et al., 2017, Li et al., 2018, Kowal et al., 2022, Xu et al., 2020, Tacchetti et al., 2017, Isik et al., 2018, Damen et al., 2022, Thoker et al., 2022, 2025a, Kumar et al., 2024].Entries share the video pretraining recipe and are paired with a single-frame DINOv3 appearance control; image-path entries initialize from DINOv3 ViT-B/16, while VideoMAE-style 3D ViT entries train from scratch.Source: §2 · pp. 3–4

3 Method

TT-VidT pairs a per-frame spatial encoder with compact temporal transfer and reconstructs target frames from a first-frame appearance anchor and motion tokens. TT1D mixes motion tokens across time, while TT3D also exposes downsampled spatial context to cross-frame attention.

3 Method

  • TT-VidT combines a DINOv3-initialized 2D spatial path, a compact Temporal Transfer pathway, and Diff Compression, which reconstructs later frames from a first-frame anchor and per-target motion tokens.The encoder, Temporal Transfer Layer, decoder, and motion-token embeddings are trained jointly under the matched recipe.Source: §3 · p. 4, §3.3 · p. 5

3.1 Temporal Transfer

  • TT1D uses motion tokens to summarize each frame’s spatial context and exchange information across frames, while the spatial path processes frames independently.Its cross-frame attention sequence has length 64, compared with 2048 for attention over full spatial features, keeping the matched comparison setting feasible.Source: §3.1 · p. 4

3.2 TT3D: Joint Spatial-Temporal Mixing

  • TT3D adds a coarse spatial view to cross-frame attention, jointly mixing motion tokens with 16 downsampled spatial tokens per frame.The resulting joint sequence has length 192, and its downsampling does not alter the full first-frame appearance features used by the decoder.Source: §3.2 · pp. 4–5

3.3 Diff Compression

  • Diff Compression conditions target-frame reconstruction on the first-frame spatial anchor and the target frame’s learned motion embedding, rather than an autoregressive frame chain.On the same encoder, MAE-Diffusion underperforms Diff Compression; the sweep also finds that neither TT3D with MAE nor Diff Compression with a strong full-3D encoder alone reaches the paired method’s motion-heavy regime.Source: §3.3 · p. 5

4 Experiments

Under a matched pretraining recipe, TT3D with Diff Compression shows a joint motion-sensitive advantage that neither component achieves alone. In the final comparison, TT-VidT leads four motion-heavy evaluations simultaneously, while the experiments also show decoder, efficiency, and task-scope trade-offs.

4.1 Setup

  • The 24-entry sweep uses a shared 8-epoch recipe at roughly 170M–190M encoder scale, but its no-augmentation condition can handicap DisMo’s native motion-preserved training.The setup uses OpenVid and Moments-in-Time v2; the authors report DisMo with dual augmentation separately.Source: §4.1 · p. 6

4.2 Synergy: Architecture-Objective Interaction

  • TT3D plus Diff Compression reaches the sweep’s highest no-augmentation scores on Jester (53.89) and SSv2 (18.28); replacing either component sharply lowers both scores.The authors interpret the paired drops as an interaction rather than a generic effect of either component alone.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 Decoder Ablation

  • Video-pretrained decoder S reaches 73.25 on Jester and 25.92 on SSv2, while the same initialization at decoder L reaches 34.37 and 12.15; decoder size effects are scope-dependent.The selected S, video-pretrained decoder also yields 37.47 on ARID and 25.15 on HMDB.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 Efficiency: Encoder FLOPs

  • TT3D uses 456.1 GF encoder-only FLOPs, about half of DisMo-2D-3D and less than half of VideoMAE-3D under the stated T=8, 256^2 setting.Its temporal-attention sequence has 192 tokens rather than 2048 for full spatial-temporal attention.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 Motion Probe and Controls

  • Under motion inversion, TT-VidT follows the altered motion on nearly every clip, while baselines retain some original answers; the single-frame DINOv3 control never follows reversal.The shuffle comparison distinguishes frame-order dependence from faithful reading of motion direction.Source: §4.5 · pp. 8–9, Table 4 · p. 8

4.6 Final Comparison

  • TT-VidT is the only compared method to lead Jester, Something-Something V2, ARID, and Diving48 fine-tuning simultaneously, scoring 73.25, 25.92, 37.47, and 18.63, respectively.These four datasets are where motion features add the most value relative to a single-frame appearance baseline.Source: §4.6 · p. 9, Table 5 · p. 9
  • After 30-epoch end-to-end fine-tuning, TT-VidT remains 21.0 points ahead of the strongest baseline on Jester and 12.9 points ahead on SSv2.The baselines converge near 51.5 on Jester, below TT-VidT’s frozen 73.25, while TT-VidT barely changes from its frozen probe.Source: §4.6 · p. 9, Table 6 · p. 9
  • The final profile has task-specific exceptions: VideoMAE leads HMDB51 and EPIC-Kitchens verb classification, DisMo leads IARD, and V-JEPA 2 leads EPIC-Kitchens anticipation.V-JEPA 2’s native large-scale recipe is outside the matched protocol; its anticipation result is 23.07 on untrimmed observation windows.Source: §4.6 · p. 10

5 Conclusion

TT-VidT separates static appearance in a first-frame anchor from temporal change routed through a compact path, with Diff Compression reconstructing target frames from the anchor and motion tokens. In a controlled sweep, the TT3D–Diff Compression pairing is the one that reaches the strong-motion regime, and TT-VidT leads the final comparison on four motion-heavy evaluations.

5 Conclusion

  • TT-VidT alone leads Jester, Something-Something V2, ARID, and Diving48 fine-tuning simultaneously in the final comparison.The sweep finds that TT3D with MAE and Diff Compression with ViT3D or DisMo remain below TT-VidT on Jester and Something-Something V2.Source: §5 · p. 10
  • The strong-motion regime comes from pairing TT3D with Diff Compression, not from either component independently.The decoder ablation selects the video-pretrained decS setting.Source: §5 · p. 10
  • The compact temporal path makes frame-to-frame change cheap and salient while the wide first-frame anchor supplies appearance for reconstruction.Diff Compression reconstructs target frames from the anchor and motion tokens, so the tokens carry information not recoverable from the anchor alone.Source: §5 · p. 10
  • TT3D uses a 192-token temporal-attention sequence and 456.1 GF encoder FLOPs, roughly half those of parameter-comparable full video encoders.Source: §5 · p. 10
  • The evidence is scoped to one roughly 1.7M-clip mixture, single-run sweep cells with multi-seed checks on canonical cells, and one roughly 170M–190M encoder scale.The study did not test TT3D with DisMo-style dual augmentation, alternative appearance probes, or partial spatial-encoder unfreezing.Source: §5 · p. 10

A Appearance-vs-Motion Diagnostic

The diagnostic compares each dataset’s best motion-aware score with a single-frame DINOv3 appearance reference to indicate where temporal information may add value. Its results are suggestive rather than a strict separation of appearance and motion, and the profile has distinct boundary cases.

A Appearance-vs-Motion Diagnostic

  • Jester and ARID score above the single-frame DINOv3 baseline, while Diving48 is below it in the frozen-probe view but reaches 18.63 after fine-tuning versus 8.12 for DisMo with dual augmentation.Something-Something V2 is absent from the plot but shows the same pattern in the architecture-objective sweep.Source: A Appearance-vs-Motion Diagnostic · p. 16
  • HMDB51 is appearance-discriminable, while IARD’s apparent appearance-versus-motion position is dominated by actor identity and requires a separate identity-control check.Table 7 reports frozen kNN@20 identity classification across five IARD actors, with lower accuracy indicating less retained per-frame identity.Source: A Appearance-vs-Motion Diagnostic · p. 16, Table 7 · p. 16
  • Figure 3 plots single-frame DINOv3 attentive accuracy on the x-axis and the best motion-model attentive score on the y-axis; points above the diagonal indicate apparent temporal value.The experiment is a quick diagnostic, not a formal benchmark or proof of motion understanding.Source: Figure 3 · p. 16, A Appearance-vs-Motion Diagnostic · p. 16
  • TT-VidT’s IARD identity accuracy is 70.99 versus the 100.00 appearance reference, while V-JEPA 2, DisMo with dual augmentation, and VideoMAE score 98.79, 99.56, and 98.90.The passage interprets this as TT-VidT retaining less per-frame identity than the broader baselines.Source: A Appearance-vs-Motion Diagnostic · p. 16
  • EPIC-Kitchens mixes verb, noun, and scene signals, and appearance-heavy or anticipation-heavy settings can favor other operating points.The authors frame these observations as boundaries on TT-VidT’s motion-priority claim.Source: A Appearance-vs-Motion Diagnostic · p. 16

B Architecture Sizing

The appendix reports analytical encoder and decoder sizing alongside pretraining details. It describes the conditions used to compare configurations and identifies the default appearance-anchored decoder setup.

B Architecture Sizing

  • TT-VidT in TT1D or TT3D form remains within roughly 1.2× the T=8 per-frame DINOv3 cost; Table 8 reports encoder parameters and FLOPs at T=8 and 256×256.V-JEPA2 uses T=16 raw frames and produces eight latent frames; the table lists both for transparency.Source: B Architecture Sizing · p. 17, Table 8 · p. 17
  • Table 9 covers decoder sizes S, B, and L, with or without first-frame cross-attention; decS with cross-attention is the main-experiment default.For Diff Compression, the decoder cross-attends to the full first-frame anchor.Source: B Architecture Sizing · p. 17, Table 9 · p. 17

C Encoder Pretraining

  • Encoder pretraining uses OpenVid-1M and Moments-in-Time, sampling eight frames at 6 FPS and mapping them through a frame VAE before reconstruction.The same recipe is applied across the architecture-objective sweep, so main-table row differences reflect architecture and objective rather than schedule.Source: C Encoder Pretraining · p. 17

D Decoder Pretraining

  • The decoder is pretrained separately for checkpoint reuse, using either ImageNet-1k repeated across the frame budget or the encoder’s video corpus.Decoder-only pretraining uses latent regression as a lightweight surrogate for the Diff Compression reconstruction target.Source: D Decoder Pretraining · p. 17

E Frozen-Probe Evaluation

Frozen-probe evaluation uses six feature probes under dataset-specific training schedules, while Diving48 is also evaluated with end-to-end fine-tuning. The appendix provides the associated hyperparameter tables.

E Frozen-Probe Evaluation

  • Frozen-probe evaluation reports six probes over mean-pooled features, including kNN, linear, MLP, and attentive variants.Table 12 summarizes the evaluation hyperparameters.Source: E Frozen-Probe Evaluation · p. 17, Table 12 · p. 18
  • Small datasets use batch 64 for 100 epochs and large datasets use batch 256 for 20 epochs; feature extraction uses T=16 for Diving48 and T=8 by default.DINOv3 and V-JEPA2 use ImageNet normalization, while the paper’s models use [0.5, 0.5, 0.5] normalization.Source: E Frozen-Probe Evaluation · p. 17

F End-to-End Finetuning

  • Diving48 end-to-end fine-tuning runs for 50 epochs with a 1e−5 encoder learning rate and 1e−3 head learning rate, using light resize-and-flip augmentation.The head is attentive pooling followed by a two-layer MLP; Table 13 lists hyperparameters.Source: F End-to-End Finetuning · p. 18

G Total Compute Footprint

The study reports compute costs across its components while preserving hardware-specific accounting. Under the matched recipe, TT3D with Diff Compression performs strongly on motion-heavy tasks across probes.

G Total Compute Footprint

  • Compute reporting separates encoder pretraining, decoder pretraining, and total study costs, without normalizing costs across GPU types.Encoder pretraining ran on B200 GPUs and decoder pretraining on H100 GPUs; Table 14 reports median encoder cost per architecture using decS.Source: G Total Compute Footprint · p. 18, Table 16 · p. 19, Table 14 · p. 19

H Complete Results

  • All pretrained runs follow the matched recipe unless a table specifies a change, and frozen evaluation uses six probes.The probes include kNN@20, pooled-token linear and MLP heads, and the attentive probe.Source: H Complete Results · p. 18

H.1 Architecture-Objective Sweep under Every Probe

  • TT3D with Diff Compression is the strongest canonical cell on Jester, SSv2, ARID, and Diving48 finetuning, with Jester and SSv2 leads holding across all six probes.Figure 4 tracks canonical cells across probes; the section also reports every cell under every probe.Source: Figure 4 · p. 19, §H.1 · p. 19

H.2 Decoder Ablation under Every Probe

Decoder ablations favor video-pretrained decoders of size S or B on every task. Additional probes and controls examine motion sensitivity, recipe variants, attribution, and finetuning.

H.3 Training-Recipe Variants

  • Dual augmentation raises Jester scores for every architecture and raises DisMo's score the most, motivating its use in the final DisMo comparison.The variants also include decoder changes and adding Kinetics-710 to the pretraining mixture.Source: §H.3 · p. 19, Table 26 · p. 23, Table 27 · p. 24

H.2 Decoder Ablation under Every Probe

  • The video-pretrained decoder outperforms the ImageNet-pretrained decoder on motion-heavy tasks at every size, while size S or B is best on every task.Figure 6 presents the decoder ablation across tasks.Source: §H.2 · p. 19, Figure 6 · p. 22

H.4 Final Comparison and Every Run

  • The final-comparison tables cover six frozen probes, two V-JEPA 2 frame rates, and every pretrained run of this work.The single-frame DINOv3 row is an attentive-probe appearance reference.Source: §H.4 · p. 19, Table 29 · p. 24, Table 30 · p. 25

I.3 End-to-End Finetuning

  • On Jester, baselines gain 12 to 35 points with end-to-end finetuning and converge near 51.5, while TT-VidT stays at frozen accuracy and leads by 21.0 points.On SSv2, TT-VidT leads by 12.9 points.Source: §I.3 · p. 25

I.1 Motion-Inversion Probe

  • TT-VidT keeps the original class on at most 1% of clips on every motion-inversion axis, versus 9% to 43% for every baseline.The single-frame control scores 0.0 flip and 100.0 stay under reversal; the flip test indicates only TT-VidT reads motion direction faithfully.Source: §I.1 · p. 22, Table 32 · p. 26

I.2 Attribution Controls

  • The learned motion token improves on an explicit feature-difference token across all five datasets, while DINOv3 initialization alone does not create the motion regime.The 3D ViT initialized from DINOv3 lands on the scratch ViT3D + MAE row for every motion-heavy dataset and moves only IARD.Source: §I.2 · pp. 24–25, Table 34 · p. 26

I.4 Seeds and Training Budget

Seed and continuation tests support the reported ordering under the matched budget, while the paper frames its broader impact as methodological and notes deployment concerns.

I.4 Seeds and Training Budget

  • From 8 to 15 epochs, the ordering is unchanged, the Jester gap stays above 31 points, and each TT3D interval after epoch 6 changes SSv2 by less than one point.The paper states its conclusions for the matched 8-epoch budget.Source: Table 35 · p. 26, Table 37 · p. 26, §I.4 · p. 26
  • On Jester, SSv2, and ARID, the worst TT3D + Diff Compression measurement exceeds the best ViT3D + MAE measurement across three pretraining and three probe seeds.On IARD, variance comes mostly from which actor is held out, and models reach similar levels under one fixed actor split.Source: Table 36 · p. 26, §I.4 · p. 26

J Broader Impact

  • The paper characterizes its positive impact as methodological and says deployment should account for privacy, consent, fairness across capture conditions, and domain-specific failure modes.It describes the work as a representation-learning study rather than a deployed system.Source: J Broader Impact · pp. 26–27

K Limitations and Future Work Discussion

The authors bound their conclusions by the study's scale and statistical coverage, and report using existing datasets under their original access terms.

K Limitations and Future Work Discussion

  • The study uses one matched pretraining mixture, one roughly 170M∼190M encoder scale, and mostly single-seed runs, so persistence at larger scales is untested.Future work could test longer pretraining, stronger data mixtures, multi-seed evaluation, dual augmentation, and broader task settings.Source: K Limitations and Future Work Discussion · p. 27

L Data, Licenses, and Release

  • The work introduces no new data source and does not redistribute Moments-in-Time v2 or downstream benchmarks.OpenVid-1M is distributed for research use under CC-BY-4.0, subject to upstream video licenses.Source: L Data, Licenses, and Release · p. 27
Loading 2609.33419v1…