PaperlayerView exact revision on arXiv

Source-linked AI summary

You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences

Ninad Daithankar, Alexi Gladstone, Yann LeCun, Heng Ji

arXiv:2606.15956v1cs.CVcs.AIcs.LG

TL;DR

Existing self-supervised visual representation learning relies on strong inductive biases, motivating methods with fewer assumptions. TDV learns from video using temporal differences and matches state-of-the-art recipes on dense spatial tasks.

  • Problem

    Existing self-supervised methods rely on strong inductive biases, motivating the question of what weaker assumption can still provide sufficient learning signal.

  • Method

    TDV jointly trains frame and motion encoders so the current-frame representation plus frame-delta representation matches the next-frame representation.

  • Results

    TDV achieves dense spatial features comparable to modern approaches using stronger inductive biases, with comparable or sometimes improved dense/spatial task performance.

  • Takeaways & Limitations

    TDV lays the groundwork for potentially more scalable visual representation learning without strong inductive biases.

  • Takeaways & Limitations

    TDV does not achieve state-of-the-art results across the board and performs poorly on semantic benchmarks.

Abstract

from arXiv · show

Progress in AI has largely been driven by methods that assume less. As compute and data increase, approaches with weaker inductive biases generally outperform those with stronger assumptions. This is particularly characteristic of the field of Visual Representation Learning, where approaches have gone from being dominated by Supervised Learning, to Weakly Supervised Learning, to the now widespread success of Self-Supervised Learning without human labels. Yet, even modern Self-Supervised Learning approaches still depend on strong inductive biases such as augmentations, masking, or cropping. If this trend holds, even these remaining biases should become bottlenecks at scale -- and our experiments confirm this: the optimal strength of inductive biases decreases as data grows. This motivates the search for approaches that rely on fewer assumptions. To this end, we introduce Temporal Difference in Vision (TDV), a new paradigm for self-supervised learning from video that avoids existing inductive biases, relying instead on a causal assumption that the past causes the future. TDV functions by jointly training an image encoder and a motion encoder so that the current frame's representation plus the encoded motion equals the next frame's representation. Despite not leveraging any strong inductive biases, TDV matches state-of-the-art recipes on dense spatial tasks, laying the foundation for representation learning without strong assumptions.

1 Introduction

The introduction argues that visual representation learning improves as inductive biases weaken, especially with increasing data scale, motivating TDV’s causal alternative. TDV learns representations from consecutive video frames by combining current-frame and motion embeddings to predict the next frame, achieving dense spatial features on par with stronger-bias approaches.

  • Visual representation learning has progressed from label- and convolution-based assumptions toward approaches using progressively weaker inductive biases, including global attention and self-distillation with ViTs.
  • As data scale increases, weaker inductive biases asymptotically outperform stronger ones, reinforcing the importance of minimizing assumptions.
  • TDV replaces existing strong inductive biases with the causal assumption that the immediate future is generally predictable from the past.
  • TDV jointly trains frame and motion encoders so the current frame’s embedding plus the frame-delta embedding matches the next frame’s embedding.Consecutive frames are close in time, making frame differences intrinsically low-rank and encouraging compact spatial-change representations.
  • TDV achieves dense spatial features on par with modern approaches that use stronger inductive biases.

2 Related Work and Background

Self-supervised representation learning has progressed from pixel-space autoencoding toward latent-space prediction with weaker inductive biases. Related work also explores temporally adjacent video frames as paired views, alongside contrastive and temporal-difference approaches.

  • Self-Supervised Representation Learning: Self-supervised representation learning aims to learn representations without labels, initially using autoencoders to reconstruct inputs in pixel space.
  • Joint Embedding Predictive Architectures: The field shifted from raw-pixel reconstruction toward JEPAs, which predict in latent space and abstract away irrelevant, unpredictable information.This favors modeling more important information with weaker inductive biases.
  • Contrastive Objectives: Early JEPA approaches used contrastive objectives that separate representations of distinct images and align augmented views, imposing a relational prior.The prior assumes randomly sampled images should be dissimilar in representation space.
  • Temporal Video Views: Because modern self-distillation still uses cropping, masking, and augmentations, researchers have explored temporally adjacent video frames as alternative paired views.Related work includes temporal-difference features trained to disentangle task-relevant motion from noise by minimizing mutual information.

3 TDV Approach

TDV replaces strong augmentation and masking assumptions with causality, using video’s temporal structure to learn representations without discarding temporal changes. It jointly encodes frames and motion so the current representation plus motion predicts the next representation, while self-distillation prevents collapse.

  • Motivation: TDV uses causality—the principle that the past is predictive of the future—as its central learning assumption, applied to video because consecutive frames provide temporal structure.The approach treats causality as an exactly correct learning signal that does not impose invariance constraints or require discarding variation.
  • Motivation: Removing DINO’s augmentations degrades KNN performance on SSV2 and eventually causes representation collapse, whereas TDV avoids collapse without these inductive biases.Table 1 attributes the degradation to progressively removing random global and local crops and other augmentations.
  • Architecture: A motion encoder maps inter-frame change into latent motion, and TDV predicts the next representation through additive composition: current representation plus encoded change.The decomposition assigns frame content to the frame encoder and temporal evolution to the motion encoder; consecutive-frame changes are typically low-rank.
  • Architecture: The frame encoder maps each frame x_t to token embeddings z_t = f_θ(x_t) ∈ R^n×D, defining the representation space for temporal prediction.n counts spatial patches plus a [CLS] token, and D is the embedding dimension.
  • Preventing collapse: TDV prevents trivial constant representations with a teacher-student self-distillation loss using student and teacher projection distributions over [CLS] and patch tokens.The teacher is updated slowly via EMA, while the cross-entropy objective penalizes collapse and encourages spatially consistent representations.

4 Experimentation

Experiments show that weaker masking assumptions become more effective as ImageNet data scales, while TDV learns competitive spatial and temporal representations without explicit semantic objectives. Ablations identify the motion encoder and MSE loss as critical for stable training and performance.

  • Inductive-bias scaling: As ImageNet data increases from 0.1% to 100%, the optimal masking strength decreases: 50% performs best at 0.1%, while 30% eventually surpasses it and 10% approaches it.The experiments compare data subsets of 0.1%, 1%, 10%, and 100%.
  • Evaluation setup: TDV is pretrained on SomethingSomethingV2 and evaluated with pretrained backbones on downstream spatial and temporal tasks.SSV2 is used because it provides well-defined motion data and is a standard video benchmark.
  • Spatial representation: TDV achieves semantic-segmentation results comparable to DINO and iBOT, trailing slightly on both mIoU and mAcc despite lacking an explicit semantic objective.TDV’s features remain spatially coherent enough for a segmentation head to leverage, although its [CLS]-token attention is less object-focused.
  • Ablations: Removing the motion encoder or MSE loss causes training to collapse, establishing both as critical components of TDV.The motion encoder supplies the temporal signal, while MSE supervises prediction of meaningful representation changes.
  • Ablations: Including the [CLS] token in cross-attention and DINO loss improves performance, while centering the teacher output matters more than temperature sharpening.These choices support grounding motion predictions in global scene representations and preventing representation collapse.

5 Future Works and Broader Impact

TDV’s weak inductive biases and joint frame–motion encoder design motivate future directions centered on scaling and reduced assumptions. By avoiding augmentations, masking, contrastive objectives, and other strong inductive biases, TDV is positioned for stronger asymptotic performance than existing recipes.

  • Future work: TDV’s weak inductive biases and joint frame–motion encoder design open several directions for future work.The paper identifies the design itself as a basis for extending the approach.
  • Scaling: Deep learning approaches with weaker assumptions tend to scale more favorably with compute and data.This motivates investigating TDV as a lower-assumption alternative as resources increase.
  • Reduced assumptions: TDV avoids augmentations, masking, contrastive objectives, and other strong inductive biases.Its training paradigm is presented as relying on fewer assumptions than existing recipes.
  • Broader impact: TDV is positioned for stronger asymptotic performance than existing recipes.This position follows from its use of weak inductive biases and the stated scaling trend.

6 Limitations and Conclusion … A.2 Neuroscientific Intuitions

TDV is presented as a first unsupervised video-representation approach that avoids strong inductive biases, while remaining limited on some evaluations. The appendix frames this minimal-prior approach as temporal compression with connections to biological vision theories.

  • 6 Limitations and Conclusion: TDV matches existing approaches on dense spatial tasks but does not achieve state-of-the-art results across the board.The authors expect future work to close the remaining gap.
  • 6 Limitations and Conclusion: TDV is proposed as the first approach to learn video representations without supervision, raw pixel-space reconstruction, or strong inductive biases.The conclusion reports comparable or sometimes improved dense/spatial performance against recipes such as DINO and iBOT.
  • A Additional Intuition: TDV avoids raw reconstruction, multimodal alignment, hand-crafted pretext tasks, contrastive learning, clustering, augmentations, explicit invariances, redundancy reduction, cropping, and masking simultaneously.This list defines the approach’s unusually broad avoidance of common inductive biases.
  • A.1 Broader Philosophical Intuitions: The authors argue that the ideal inductive bias is the weakest one that still enables learning within a single lifetime of experience.It should bootstrap learning from finite data without bottlenecking what can ultimately be learned.
  • A.1 Broader Philosophical Intuitions: TDV can be understood as compressing temporal experience: the motion encoder captures frame changes, while the frame encoder captures information needed to predict the future.This formulation imposes no assumptions about augmentation invariances, negative pairs, or pixel-level fidelity.
  • A.1 Broader Philosophical Intuitions: TDV also admits an interpretation as a small latent action world model, with temporal differences functioning as latent actions that make future frames predictable.The motion encoder captures the temporal differences that cause future-frame predictability.
  • A.2 Neuroscientific Intuitions: TDV loosely mirrors biological vision theories through predictive coding and a dorsal/ventral-like factorization of frame and motion encoding.The MSE objective predicts the next-frame embedding, while separate encoders operate on the current frame and its temporal difference.
  • A.2 Neuroscientific Intuitions: TDV’s [CLS] attention is less object-focused than DINO’s and iBOT’s because TDV does not use a dominant [CLS]-token objective during pretraining.The figure visualizes attention maps for ViT-B models pretrained on SSv2 across three ImageNet images.

B Additional Experiments

Additional experiments show that TDV remains behind state-of-the-art recipes on semantic representation, while its patch features exhibit clean, spatially coherent object structure. The semantic gap is attributed to TDV’s lack of strong inductive biases.

  • Semantic benchmarking: TDV currently lags behind state-of-the-art recipes on semantic representation benchmarks.The reported explanation is TDV’s lack of strong inductive biases.
  • Patch-feature visualization: TDV’s patch-level features produce clean, spatially coherent feature maps aligned with object boundaries.This is shown through RGB visualizations of the top-3 PCA components from ViT-B models pretrained on SSv2.
  • Patch-feature visualization: TDV often yields better object-aligned patch-feature maps than DINO and iBOT.Similar colors in the PCA visualizations indicate patches with similar representations.

B.1 Experiments We Tried that Didn’t Work · C TDV Details

TDV proved difficult to train without strong inductive biases, and many attempts to scale data, combine datasets, alter conditioning, or add augmentation failed or destabilized learning. The experiments support retaining natural temporal differences, sufficient motion-encoder capacity, and stable EMA supervision rather than imposing additional mechanisms.

  • B.1 Experiments We Tried that Didn’t Work: TDV training was generally difficult because the method does not use strong inductive biases.The authors document design choices and training strategies that they explored but found ineffective.
  • B.1 Experiments We Tried that Didn’t Work: Larger video datasets did not improve representations: Ego4D produced noisy RGB differences, while FineVideo’s abrupt cuts made temporal differences unreliable.Ego4D contains highly variable motion, whereas FineVideo contains many abrupt scene cuts.
  • B.1 Experiments We Tried that Didn’t Work: Mixing SSv2, Kinetics-400, and Ego4D degraded ImageNet KNN relative to training on individual datasets alone.The authors attribute this to heterogeneous motion statistics across the datasets.
  • B.1 Experiments We Tried that Didn’t Work: FiLM, AdaLN, AdaLN-Zero, and Gated AdaLN initially achieved 2x better KNN accuracy than standard cross-attention but later plateaued or collapsed.These mechanisms were alternatives to conditioning the motion encoder on frame-encoder tokens.
  • B.1 Experiments We Tried that Didn’t Work: RGB difference thresholding did not improve results, because near-static frames may calibrate the motion encoder toward producing near-zero deltas.The thresholding strategy skipped the motion-encoder pass for frames with small RGB differences.
  • B.1 Experiments We Tried that Didn’t Work: An auxiliary motion embedding divergence loss destabilized training by causing frame-representation variance to explode without improving KNN or downstream performance.The loss was intended to push consecutive frame embeddings farther apart.
  • B.1 Experiments We Tried that Didn’t Work: iBOT-style masking collapsed above 10% masking, while lower rates did not meaningfully improve performance over the unmasked baseline.Masking corresponding patches in the current frame and RGB-difference input created the main consistency challenge.
  • B.1 Experiments We Tried that Didn’t Work: DINO-style augmentations consistently caused collapse, whereas identical teacher and student sharpening outperformed asymmetric teacher sharpening.Strong independent augmentations alter the temporal relationship required by TDV’s causal motion objective.

C.1 Training Recipe … D Experimental Details

The appendix specifies TDV’s gradient-and-EMA training procedure, shared pretraining setup, and temporal-difference sampling choices. It also details the SSv2-based comparison with DINO and iBOT and the implementation settings used for these recipes.

  • C.1 Training Recipe: TDV updates the student frame and motion encoders by gradient descent, while the teacher frame encoder is updated only through EMA.The teacher frame encoder receives no gradients.
  • C.2 Pretraining Setup: TDV, DINO, and iBOT are pretrained on SSv2, a dataset of approximately 220,000 short egocentric hand-object interaction videos.SSv2 is used for initial experimentation because of its manageable size and consistent motion quality.
  • C.2 Pretraining Setup: All three models use ViT-S and ViT-B architectures and are trained for 20 epochs, corresponding to around 200,000 steps.Results are reported from the final checkpoint.
  • C.3 Hyperparameters: Shared batch size, optimizer, learning-rate schedule, and EMA momentum settings are kept identical across methods wherever possible for fair comparison.Customized augmentations and temperature-sharpening hyperparameters remain at default values to maximize each recipe’s SSv2 performance.
  • C.3 Hyperparameters: TDV uses the RGB difference ∆x_t = x_t+1 − x_t between temporally adjacent frames sampled at a fixed stride.The stride determines how much motion is visible in the difference images.
  • C.3 Hyperparameters: Too small a temporal stride yields near-zero differences for slow-moving scenes, whereas too large a stride creates large, incoherent pixel jumps from drastic object-position changes.These stride effects motivate careful temporal-sampling choices for TDV pretraining.

D.1 Philosophical Backing Experiments

The experiments test whether weaker assumptions perform better as data scales by varying ImageNet-1k data size and augmentation strength. They use DINO with iBOT-style patch masking, while retaining random resized cropping to prevent collapse under low augmentation.

  • Experimental setup: Models are trained on ImageNet-1k subsets to measure performance across data scale and augmentation strength.The experiments empirically test whether weaker assumptions improve as data scales.
  • Experimental setup: The study uses the DINO codebase augmented with iBOT-style patch-level masked prediction.This setup varies masking ratio continuously as a proxy for assumption strength.
  • Experimental setup: Random resized cropping is retained to prevent collapse under low-augmentation regimes.

D.2 Semantic Evaluations

The section evaluates TDV’s semantic representations, expecting limited semantic performance because its default recipe uses no augmentations, explicit invariances, or strong inductive biases. It reports frozen-probe action recognition on Something-Something V2 and online ImageNet KNN as a lightweight pretraining signal.

  • Semantic representation evaluation: TDV’s default recipe is expected to yield less-semantic representations because it omits augmentations, explicit invariances, and strong inductive biases.The experiments confirm that semantic-task performance without these inductive biases is not great.
  • Action recognition: Action recognition uses a frozen encoder and a task-specific linear probe on concatenated [CLS] features from 8 uniformly sampled Something-Something V2 frames.The evaluation reports validation-set Top-5 accuracy following V-JEPA’s frozen evaluation protocol.
  • KNN on ImageNet: Online ImageNet KNN Top-5 accuracy monitors representation quality and detects pretraining collapse without the cost of full downstream evaluation.The method uses k = 20 nearest neighbors on [CLS] features from ImageNet training-subset images after each epoch.

D.3 Downstream Spatial Evaluations · D.4 Compute Resources

The paper evaluates TDV on dense spatial tasks using frozen-backbone semantic segmentation and end-to-end optical-flow and stereo-depth fine-tuning. Pretraining uses two NVIDIA H100 GPUs for approximately 48 hours per 20-epoch run, while downstream evaluations use task-specific compute budgets.

  • D.3 Downstream Spatial Evaluations: Semantic segmentation uses frozen ViT backbones with only a UperNet head trained for 320,000 steps on ADE20K and Cityscapes.mIoU and mAcc are reported on the respective validation sets from the final checkpoint, using the same configuration for all baselines.
  • D.3 Downstream Spatial Evaluations: Optical flow is evaluated by end-to-end fine-tuning on FlyingChairs, FlyingThings3D, and MPI-Sintel, with MPI-Sintel clean and final validation EPE reported.The standard CroCo setup uses Midway Networks two-frame feature fusion followed by a DPT prediction head; lower EPE is better.
  • D.3 Downstream Spatial Evaluations: Using TDV motion-encoder representations directly with the DPT head yields EPE (clean) 14.53 and EPE (final) 14.52 for the TDV base variant.This variant removes the Midway Networks decoder and supplies intermediate motion-encoder embeddings to the DPT head.
  • D.3 Downstream Spatial Evaluations: Stereo depth uses end-to-end fine-tuning on SceneFlow (final), evaluated on its final validation set with average disparity error and bad pixel rates at 0.5px and 1px thresholds.The evaluation uses the CroCo Midway Networks two-frame decoder and DPT head, with all models fine-tuned for 32 epochs.
  • D.3 Downstream Spatial Evaluations: The TDV motion-encoder stereo variant produces average disparity error 6.93 for the TDV base variant and 7.23 for the TDV small variant.Intermediate motion-encoder representations are provided as additional input to the DPT head.
  • D.4 Compute Resources: TDV, DINO, and iBOT pretraining runs use 2 NVIDIA H100 GPUs with 80 GB GPU memory each, train for 20 epochs on SSv2, and take approximately 48 hours per run.The reported hardware and duration apply to all listed pretraining runs.
  • D.4 Compute Resources: Semantic-segmentation fine-tuning uses a single NVIDIA H100 GPU for 320,000 steps and takes approximately 20 hours per run, while optical-flow fine-tuning uses 2 NVIDIA H100 GPUs with BF16 and takes approximately 48 hours per run.The segmentation runs cover ADE20K and Cityscapes; optical-flow runs cover FlyingChairs, FlyingThings3D, and MPI-Sintel.
  • D.4 Compute Resources: Stereo-depth fine-tuning on SceneFlow uses the same 2 NVIDIA H100 GPUs with 80 GB GPU memory each for 32 epochs and takes approximately 16 hours per run.This compute configuration corresponds to the SceneFlow stereo-depth evaluation.
Loading 2606.15956v1…