PaperlayerView exact revision on arXiv

Source-linked AI summary

I Have a Stream: Making Self-Supervised Learning Work on Continuous Video

Ivan Martinović, Lukas Knobel, Yuki M. Asano

arXiv:2609.40333v1cs.CV

TL;DR

Self-supervised learning on continuous video streams lacks the shuffled, diverse batches common in standard image pretraining. The paper introduces WT++ and StreamMAE to study and adapt learning in this setting. StreamMAE outperforms streaming baselines, narrows the gap to i.i.d. MAE on the same video data, and scales with pretraining duration and encoder capacity.

  • Problem

    The paper asks whether self-supervised learning can work and scale on continuous video, where temporally local batches are redundant and global shuffling and multi-epoch replay are excluded.

    Source: §1 · p. 2

  • Method

    The authors construct WT++ and adapt MAE as StreamMAE, retaining reconstruction while adding stream-aware regularization and motion-biased crop selection.

    Source: §1 · p. 2

  • Results

    StreamMAE outperforms streaming baselines, narrows the gap to i.i.d. MAE on the same video data, and scales with encoder capacity and pretraining duration.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Masked reconstruction combined with stream-aware sampling and regularization is a strong foundation for self-supervised learning from continuous video streams.

    Source: §1 · p. 2

    StreamMAE does not explicitly model temporal dynamics beyond motion-biased crop selection, and its fixed frame subsampling may be suboptimal when visual-change rates vary.

    Source: B Limitations and Open Challenges · p. 18

Abstract

from arXiv · show

Self-supervised learning draws inspiration from infant visual development, yet standard training pipelines bear little resemblance to it: images are independently sampled and globally shuffled across epochs. We study self-supervised learning from continuous video streams, where frames are consumed in temporal order using strict sliding-window batches, without global reshuffling or multi-epoch replay. To this end, we construct WT++, a 95-hour urban walking-tour video dataset for streaming pretraining. Combined with a comprehensive evaluation suite we find that contrastive and distillation-based methods struggle in this setting, while MAE is more robust but still falls short of standard i.i.d. pretraining. We find that high inter-batch similarity, caused by sliding-window consumption across consecutive batches, does not explain this gap. The main challenge is high intra-batch similarity, where frames within each batch are near-duplicates. To mitigate this, we propose StreamMAE, which preserves the core MAE reconstruction objective while adapting the input pipeline with stream-aware regularization and motion-biased crop selection. StreamMAE outperforms streaming baselines, matches i.i.d. MAE trained on the same video data, remains competitive with ImageNet-pretrained MAE, and scales positively as the pretraining stream grows from 12 to 95 hours.

1 Introduction

The paper studies from-scratch self-supervised learning on ordered video streams, where consecutive frames produce redundant batches rather than the diverse batches of shuffled image training. It introduces StreamMAE and WT++ to evaluate and address this setting.

1 Introduction

  • The study asks whether self-supervised learning can work and scale on continuous video without global shuffling, multi-epoch training, or a long-term replay buffer.Unlike standard pipelines, streaming batches contain short consecutive segments with high redundancy and small visual changes.Source: §1 · pp. 1–2
  • WT++ extends WalkingTours to 95 hours of walking-tour video, and the authors evaluate representations on close-to-domain and out-of-domain vision benchmarks..Source: §1 · p. 2
  • MoCo v3 and DINO lag behind MAE in streaming pretraining, while even streaming MAE falls short of standard i.i.d. MAE on the same data.A pre-shuffled ImageNet stream matches standard i.i.d. MAE, pointing to high intra-batch similarity—not inter-batch similarity alone—as the main challenge.Source: §1 · p. 2
  • StreamMAE outperforms streaming baselines, narrows the gap to i.i.d. MAE on the same video data, and scales with encoder capacity and pretraining duration.It preserves MAE reconstruction while adding stronger regularization, two-stage cropping, and motion-biased crop selection.Source: §1 · p. 2

2 Related Work

Prior work spans self-supervised image encoders, offline video learning, and continual or online learning. This paper distinguishes its focus as from-scratch representation learning on ordered video streams without long-term replay buffers.

2 Related Work

  • Self-supervised image encoders use contrastive, self-distillation, and masked-modeling objectives, and are commonly pretrained on shuffled image collections or offline video.These encoders transfer to classification, segmentation, depth estimation, and 3D geometry.Source: §2 · pp. 2–3
  • Video SSL methods may use temporal ordering, predictive coding, or masked video modeling, but most assume offline clips that can be sampled, shuffled, and revisited.This work instead keeps an image-based encoder and consumes frames in temporal order.Source: §2 · p. 3
  • Prior continuous-video approaches use replay or memory, pretrained initialization, or task adaptation; this work studies from-scratch ordered-stream representation learning without long-term replay.Han et al. target correlated gradient updates, but training from scratch remains substantially below pretrained-initialization settings.Source: §2 · p. 3

3 Learning from a Streaming Video

The paper defines streaming batches as temporally ordered sliding windows and constructs WT++ to support longer streaming experiments. Its dataset analysis contrasts locally consecutive frames with randomly sampled frames.

3.1 Streaming Sliding-Window Batches

  • Each batch is a B-frame window from an ordered video stream, and successive windows advance by stride s.When s < B, neighboring batches overlap by B − s frames.Source: §3.1 · p. 3
  • The setup assumes a batch of examples at each training step, rather than learning from one frame at a time..Source: §3.1 · p. 3

3.2 WT++ Dataset

  • WT++ walking-tour videos contain varied objects, lighting, viewpoints, and scene transitions, making them a testbed for learning from video streams..Source: §3.2 · p. 3
  • WT++ extends the approximately 13-hour WalkingTours dataset to 95 hours across 58 public walking-tour videos.The extension includes the original WT videos except the Wildlife safari video.Source: §3.2 · p. 4
  • The default training stream is one 12-hour video; larger experiments concatenate videos while preserving order within each video, with discontinuities at boundaries.This approximates a long continuous stream because public single-shot walking-tour videos rarely span tens of hours.Source: §3.2 · p. 4
  • Consecutive 15 FPS frames can be highly similar, while uniformly random frames from the same video are substantially more diverse.The analysis compares pairwise cosine similarities of DINOv2 features for 512 frames in each sampling condition.Source: §3.2 · p. 4, Figure 3 · p. 4

4 Towards StreamMAE

MAE is more robust than contrastive and distillation methods on streaming video, but near-duplicate frames within a batch remain a challenge relative to i.i.d. pretraining. StreamMAE keeps MAE’s reconstruction objective and adapts its training pipeline with regularization and stream-aware cropping.

4 Towards StreamMAE

  • Near-duplicate frames within batches are the main challenge for streaming MAE; high inter-batch similarity alone does not explain its gap from i.i.d. MAE.The section reports that ViT-B/16 streaming MAE underperforms by nearly 10 mIoU points on Cityscapes; pre-shuffled ImageNet-1K streaming matches standard i.i.d. MAE.Source: §4 · p. 4, §4.1 · p. 5, Table 2 · p. 5

4.1 Preliminary Analysis

  • Within the MAE family, methods closer to i.i.d. MAE in mean gradient similarity achieve stronger dense performance; StreamMAE moves gradient behavior closer and improves downstream performance.Figure 4 relates dense performance to distance from the i.i.d. MAE reference in mean consecutive-batch gradient similarity.Source: Figure 4 · p. 5, §4.1 · p. 5

4.2 Regularization under High Intra-Batch Similarity

  • Color jitter, increased drop path, and DataDrop strengthen MAE’s training pipeline for batches containing highly similar frames.Color jitter perturbs appearance statistics, drop path varies effective network depth, and DataDrop computes loss only on retained examples.Source: §4.2 · p. 6

4.3 Stream-Aware Cropping

  • Two-stage cropping first selects a fixed-size local region, then applies the standard MAE random resized crop within it.This separates the choice of where to crop from how to augment the final training view.Source: §4.3 · p. 6
  • Motion-biased selection chooses among candidate regions using average patch-level motion, then applies the standard MAE random resized crop.The patch-level map marks regions that differ from the preceding frame; selection is used with probability 0.5, otherwise standard random resized cropping is used.Source: §4.3 · p. 6, Figure 5 · p. 7

5 Experiments

Experiments evaluate streaming SSL representations across classification, segmentation, and depth estimation, then examine StreamMAE’s components, scaling, and transfer across pretraining domains.

5.1 Benchmarking Self-Supervised Methods under Streaming Pretraining

  • StreamMAE improves over streaming baselines and is comparable to same-data i.i.d. MAE; WT++95h further improves transfer, especially dense prediction.The benchmark also compares against random initialization and an iteration-matched ImageNet-1K i.i.d. MAE reference.Source: §5.1 · p. 7, Table 3 · p. 8
  • Stronger regularization and crop selection account for most dense-transfer improvement, while DataDrop gives modest gains.Motion-biased crop selection gives the largest gains on Cityscapes and KITTI.Source: §5.1 · p. 8, Figure 6 · p. 8

5.2 Scaling and Analysis

  • StreamMAE (ViT-B) improves over StreamMAE (ViT-S) and the MAE baseline while closing most of the gap to standard i.i.d. MAE.The table caption describes increasing encoder size; it does not supply individual benchmark values.Source: Table 4 · p. 9
  • DataDrop’s effect becomes negligible for ViT-B/16 on WT++50h and WT++95h, making it less central at larger scale.In the no-DataDrop setting, B = 512 corresponds to roughly 136 seconds of video at 3.75 FPS.Source: §5.2 · p. 9, Table 6 · p. 9
  • Averaging checkpoints from a ViT-B/16 WT++95h trajectory improves dense prediction while leaving ImageNet-1K accuracy essentially unchanged.The authors suggest checkpoints from different training stages may retain complementary information.Source: §5.2 · p. 9, Table 7 · p. 9
  • Longer streams generally improve performance, and new videos provide larger gains than repeated passes over WT++12h.More updates from denser temporal sampling do not improve transfer and can hurt dense prediction; continued optimization over the same stream can still help.Source: §5.2 · pp. 9–10, Table 8 · p. 9, Table 9 · p. 9

5.3 Generalization Across Pretraining Domains

  • Across HD-EPIC, CROWD, and KrishnaCAM, StreamMAE improves over streaming MAE on all four downstream tasks; it matches or exceeds same-data i.i.d. MAE on the first two domains.On KrishnaCAM, it trails i.i.d. MAE slightly on ADE20K and NYUv2 but outperforms it on Cityscapes and KITTI.Source: §5.3 · p. 10, Table 11 · p. 10
  • On KrishnaCAM, motion-biased crop selection improves ADE20K and Cityscapes by 0.6 and 0.7 mIoU, reduces KITTI RMSE by 0.063, and leaves NYUv2 unchanged.This ablation tests the crop policy under irregular camera motion beyond the walking-tour domain.Source: §5.3 · p. 10

6 Conclusion

The conclusion identifies high intra-batch similarity, rather than inter-batch similarity, as the main streaming challenge for MAE. StreamMAE adapts the pipeline while retaining MAE’s reconstruction objective, and the analysis supports that diagnosis.

6 Conclusion

  • StreamMAE retains the MAE objective while adapting the pipeline to mitigate high intra-batch similarity, and is competitive with i.i.d. MAE while scaling to longer streams.The conclusion characterizes contrastive and self-distillation methods as struggling in this setting, with masked reconstruction a more robust starting point.Source: §6 · p. 10

A.1 Intra- and Inter-Batch Similarity on WT++12h

  • The analysis finds that high inter-batch similarity does not explain the streaming MAE gap; high intra-batch similarity from near-duplicate frames is the main source of degradation.A pre-shuffled ImageNet-1K stream remains close to standard i.i.d. MAE despite high inter-batch similarity.Source: §6 · p. 10, §A.1 · p. 14
  • Pre-shuffled streaming lowers intra-batch similarity from 0.665 to 0.171 and raises ADE20K and Cityscapes mIoU by 2.1 and 2.6 for ViT-S/16, and 6.1 and 10.6 for ViT-B/16.The two streaming regimes have nearly identical inter-batch similarities of 0.999 and 1.000.Source: §A.1 · p. 14, Table 12 · p. 14

A.2 Downstream Evaluation Reliability

Reliability checks support StreamMAE’s downstream gains, while appendix analyses test batch duration, training choices, stream composition, gradient similarity, and study limitations.

A.2 Downstream Evaluation Reliability

  • StreamMAE’s gains over streaming MAE exceed observed fine-tuning variation and match or exceed same-data i.i.d. MAE across evaluated tasks.Table 13 reports repeated fine-tuning runs for fixed WT++12h checkpoints.Source: Table 13 · p. 15, §A.2 · p. 15

A.3 Sensitivity to Batch Size

  • 136, 68, and 34 seconds of video per batch correspond to B = 512, 256, and 128; performance degrades gradually and remains competitive at B = 128.This test reduces batch duration without DataDrop, using ViT-B/16 pretrained on WT++50h at 3.75 FPS.Source: Table 14 · p. 15, §A.3 · p. 15

A.4 On Baselines and DataDrop

  • A B = 2048 window with only a random 25% subset backpropagated generally matches or improves B = 512, while using the full window often hurts performance.The authors therefore report streaming baselines with DataDrop; gradient-similarity analyses use B = 512 without it.Source: Table 15 · p. 15, §A.4 · p. 15

A.5 Controlling for the Final Video

  • Appending Budapest to shorter streams does not match the full WT++50h stream, suggesting longer-stream gains also depend on preceding video.Source: Table 16 · p. 16, §A.5 · p. 16

A.6 Masking Strategy; A.7 AdamW Momentum

A.8 Additional Ablations

  • Removing color jitter and increased drop path lowers performance, while removing DataDrop remains competitive on semantic segmentation.Table 20 separately tests motion-biased crop selection with matched frames, pretraining budget, and remaining StreamMAE configuration.Source: §A.8 · p. 16, Table 19 · p. 17, Table 20 · p. 17

A.9 Longer i.i.d. ImageNet Reference

  • StreamMAE trained on WT++95h remains competitive with a longer i.i.d. ImageNet reference, particularly for ViT-S and Cityscapes transfer.The reference matches approximately 650k iterations, the training length of WT++95h streaming runs.Source: Table 21 · p. 17, §A.9 · p. 17

A.10 Cosine Similarity of Gradients from Consecutive Batches

  • Closer mean consecutive-batch gradient similarity generally tracks stronger Cityscapes and ADE20K performance, but motion-only crops break that pattern on Cityscapes.Figure 7 shows StreamMAE near i.i.d. MAE in mean similarity but with greater variance; high inter-batch similarity does not explain streaming degradation.Source: §A.10 · pp. 17–18, Figure 8 · p. 18, Figure 7 · p. 18

C WT++ dataset

WT++ is constructed as a sequence of increasingly long ordered streams, and redistribution of two videos depends on creator permission.

C WT++ dataset

C.1 Dataset release

  • For the two videos under the Standard YouTube License, direct file redistribution depends on creator permission; otherwise, only metadata and source links or compatible replacements are planned.The other 56 videos are licensed under CC BY 4.0 and are planned for redistribution with attribution.Source: §C.1 · p. 19

D MemoryStoryboard [56] Results

The MemoryStoryboard reproduction used a tuned SimCLR configuration with replay, while the accompanying analyses specify similarity measurement and stream-aware crop implementation.

D MemoryStoryboard [56] Results

  • With ViT-S/16, MemoryStoryboard used stride 2, a 2048-sample FIFO and 16384-sample reservoir, and its best tested objective was SimCLR at learning rate 10−4.The reported setup used 64 current-stream and 448 replay samples per batch; the original ResNet-50 recipe with only recent-frame memory showed signs of collapse and had no downstream results reported.Source: D MemoryStoryboard [56] Results · p. 19

E.1 Inter- and Intra-Batch Similarity Computation

  • Similarity was measured on raw samples with frozen DINOv2-L features, averaging within-batch scores and scores across consecutive batch pairs over specified sampled streams.For WT++London, batches had size 512 and stride 1; ImageNet stream comparisons used size 512 and stride 8, while i.i.d. ImageNet scores averaged 50 batches.Source: §E.1 · p. 20

E.2 Two-Stage Cropping and Motion-Biased Crop Selection

  • Motion-biased cropping uses cached patch-level frame-difference maps to score patch-aligned first-stage crops before the final random resized crop.Frames yield an 80 × 45 difference map; with DataDrop, differences are computed over the local stream window before examples are dropped, while only retained examples contribute to training loss.Source: §E.2 · p. 20

E.3 Training i.i.d. Baselines

This section specifies iteration-matched i.i.d. references, streaming SSL baseline experiments and variants, and three additional ordered pretraining streams.

E.3 Training i.i.d. Baselines

  • The i.i.d. references match the corresponding WT++London streaming runs in optimization steps, while the fixed-order ImageNet stream is also iteration-matched.The i.i.d. baselines sample batches uniformly without preserving temporal order and use the standard MAE cosine learning-rate schedule; streaming runs use constant learning rate with 5% linear warm-up.Source: §E.3 · p. 21

E.4 SSL Baselines

  • DINO, MAE, and MoCo v3 were reproduced from scratch on WT++12h with a shared single-traversal sliding-window protocol and baseline-specific configurations.The experiments swept learning rate, batch size, teacher momentum, crop configuration, and prototype count where applicable, alongside adaptations for temporal redundancy.Source: §E.4 · p. 21
  • MoCo v3 adaptations did not resolve optimization instability or downstream-transfer degradation, while DINO variants did not consistently improve transfer; Orthogonal-AdamW also gave no consistent gains.DataDrop was retained in the main streaming comparison because it matched or improved corresponding no-drop configurations.Source: §E.4 · pp. 21–22

E.5 Additional Pretraining Datasets

  • The additional ordered streams comprise 18 hours of HD-EPIC kitchen video, 14.5 hours of CROWD urban dashcam video, and 70 hours of KrishnaCAM daily-life video.Videos were kept in temporal order within videos and concatenated; KrishnaCAM used K = 2 candidate crops to reduce repeated selection during stationary activities.Source: §E.5 · p. 22

F Additional Evaluation Details

The evaluation details specify fine-tuning and validation protocols for image classification, semantic segmentation, and depth estimation.

F.1 Image Classification

  • ImageNet-1K classification follows the MAE fine-tuning protocol for ViT-S/16 and ViT-B/16, with top-1 accuracy reported on the validation set.Models are fine-tuned end-to-end for 100 epochs; MAE-based models use pooled patch tokens, while DINO and MoCo v3 use the final [CLS] token.Source: §F.1 · p. 22

F.2 Semantic Segmentation

  • ADE20K and Cityscapes segmentation follow Kerssies et al., using a linear per-patch head and end-to-end fine-tuning with validation mIoU.Training runs for 40k steps on ADE20K and 20k on Cityscapes; validation uses sliding-window inference and averages logits in overlapping regions.Source: §F.2 · p. 22

F.3 Depth estimation

  • Depth estimation follows Yang et al. [55] but uses full fine-tuning, with intermediate encoder features supplied to the DPT head.NYUv2 and KITTI are trained for 60 epochs using SiLog loss; reported metrics are mean ± standard deviation over two fine-tuning runs with different seeds.Source: §F.3 · p. 23

G Compute Resources

The compute account reports approximate H100 requirements for pretraining and downstream fine-tuning, with pretraining times excluding downstream work.

G Compute Resources

  • On 2×H100, WT++12h StreamMAE takes approximately 7 hours with ViT-S/16 and 8 hours with ViT-B/16, while MoCo v3 and DINO take approximately 18 and 25 hours.StreamMAE can run on a single H100, and longer streams typically used 4×H100 to reduce turnaround time.Source: G Compute Resources · p. 23
  • Downstream fine-tuning takes approximately 4 and 5 hours for ViT-S/16 and ViT-B/16 ImageNet-1K, respectively, on 4×H100; the full project also used unreported compute for sweeps and failed runs.The compute table reports pretraining times for WT++12h and excludes downstream fine-tuning.Source: G Compute Resources · p. 23, Table 24 · p. 23

Related papers

Loading 2609.40333v1…