PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Uncertainty as a Self-supervised Signal for Semantic Similarity Learning

Enrico Pallotta, Sina Raoufi, Lars Doorenbos, Gianni Franchi, Juergen Gall

arXiv:2609.35341v1cs.CV

TL;DR

Video similarity is difficult to label, and pretrained video features are not explicitly designed to measure it. This paper learns feature masks from T2V generation consistency, and reports improvements over original features and other subspace methods across three similarity-based tasks.

  • Problem

    Human-annotated video similarity is difficult and expensive to collect, while pretrained video representations are not explicitly designed to measure semantic similarity.

    Source: §1 · p. 1

  • Method

    The method learns masks over pretrained video embeddings from same-prompt synthetic pairs, using general-prompt consistency and specialized-prompt variability as supervision.

    Source: §1 · pp. 1–2

  • Results

    Learned feature subspaces consistently outperform original features and other subspace methods across human alignment, generated-sample OOD detection, and video retrieval.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Generative uncertainty provides a self-supervised signal for learning sparse subspaces that improve similarity tasks across different feature encoders.

    Source: §1 · p. 2

    The pipeline depends on pretrained prompt, video-generation, and encoder components, whose systematic biases or domain shortcomings may propagate into learned masks.

    Source: D LIMITATIONS · p. 22

Abstract

from arXiv · show

Evaluating semantic similarity between videos is a fundamental challenge in computer vision, essential for tasks ranging from out-of-distribution (OOD) detection to video retrieval. However, defining and labeling video similarity is notoriously difficult and expensive due to the complex spatio-temporal nature. In this paper, we propose a novel self-supervised approach that leverages generative uncertainty from text-to-video (T2V) diffusion models to learn semantic similarity without human annotations. Our method is based on the observation that T2V models produce consistent outputs for familiar concepts but exhibit high variance and uncertainty when prompted with specialized concepts. We utilize this behavior to identify stable semantic features within existing pretrained representations, such as VideoMAE and V-JEPA. Specifically, we learn a mask over these embeddings using purely generated data, encouraging the model to retain features that remain consistent across generations of general concepts while discarding those associated with generative noise or uncertainty. Experimental results across three key tasks demonstrate that our learned feature subspaces consistently outperform original pretrained features and baseline feature selection methods.

1 INTRODUCTION

Human-annotated video similarity is difficult and expensive, while pretrained video features are not explicitly optimized for semantic similarity. The paper uses differences in T2V generation consistency to learn feature subspaces that outperform original features across three similarity-based tasks.

1 INTRODUCTION

  • Pretrained video representations can contain dimensions irrelevant or detrimental to similarity tasks, and raw comparisons may align only weakly with human judgments.The paper also reports unreliable uncertainty estimates for generated videos.Source: §1 · p. 1
  • General-concept prompts tend to yield consistent generations, whereas specialized-concept prompts produce more diverse outputs, providing a signal called generative uncertainty.The paper uses this variability to identify feature dimensions useful for semantic similarity.Source: §1 · pp. 1–2
  • The method learns a mask over pretrained embeddings from same-prompt video pairs, encouraging general-prompt pairs to be more similar than specialized-prompt pairs.It uses synthetic data without human similarity annotations or predefined semantic relations between prompts.Source: §1 · pp. 1–2
  • Learned feature subspaces consistently outperform original features and other subspace methods across human alignment, generated-sample OOD detection, and video retrieval.The comparison holds across different T2V models.Source: §1 · p. 2
  • The approach learns sparse subspaces from synthetic data and improves human-similarity alignment, OOD detection, and video retrieval across feature encoders.The paper describes the method as lightweight and model-agnostic.Source: §1 · p. 2

2 RELATED WORK

Prior video-similarity work has emphasized retrieval and near-duplicate detection, while synthetic data remains comparatively unexplored for video similarity. This paper uses zero-shot T2V outputs to learn similarity without real-world video collections or manual augmentations.

Video similarity

  • Raw V-JEPA 2.1 and VideoMAE V2 representations align only partially with human judgment, and targeted masking refines them into subspaces for similarity tasks.The paper builds on these pretrained backbones.Source: §2 · p. 2
  • Self-supervised video-similarity methods include retrieval and near-duplicate approaches, but often rely on access to large video datasets.Prior approaches also use visual-stream learning or synthetic proxy tasks built from heavy data augmentations.Source: §2 · p. 2
  • Synthetic data is comparatively unexplored in video similarity; this work uses zero-shot T2V outputs to avoid real-world video collections or manual augmentations.The passage contrasts this direction with synthetic-data successes in perceptual image tasks and action recognition.Source: §2 · p. 2

Generative uncertainty

  • Earlier generative-uncertainty work measures generation uncertainty, whereas this paper uses generative variability to define a self-supervised similarity-learning objective.The authors state that they do not propose a new method for measuring generative uncertainty.Source: §2 · p. 3

3 METHOD

The method generates same-prompt video pairs from general and specialized concepts, extracts global pretrained features, and learns a mask that distinguishes their cosine similarities. It then applies the learned mask to unseen video representations for downstream similarity tasks.

3 METHOD

  • The framework generates same-prompt videos from general and specialized prompts using different random seeds, learns a mask over frozen global features, then masks unseen representations before cosine similarity.The overview reports improved human-judgment alignment, OOD detection, and video retrieval.Source: Figure 1 · p. 3

3.1 SYNTHETIC DATASET GENERATION

  • The general prompt set contains 100 VBench prompts chosen to represent concepts Wan2.1 can synthesize with high fidelity and consistency.The exact Wan2.1 training corpus is proprietary; the authors cite its documented broad domains and VBench performance as motivation for this choice.Source: §3.1 · p. 3
  • The specialized set contains 100 Gemini-generated prompts from uncommon or niche domains, matched to general prompts in length, syntax, and descriptive granularity.The domains include medical surgery and microscopy.Source: §3.1 · p. 4
  • For each of 200 prompts, three distinct videos were synthesized by varying the initial random seed, yielding 600 videos; general prompts proxy semantic consistency, while specialized prompts signal generative divergence.Source: §3.1 · p. 4

3.2 FEATURE LEARNING VIA GENERATIVE CONSISTENCY

  • A pretrained ViT-based encoder's patch tokens are globally average-pooled into one representation to prioritize global semantic similarity over localized cues.The resulting representation is z = E(v) in R^D.Source: §3.2 · p. 4
  • A learnable soft mask is trained with MSE so same-general-prompt pairs target cosine similarity 1 and same-specialized-prompt pairs target 0; early stopping uses their average-similarity margin, and threshold 0.5 binarizes the mask.Both soft and binarized masks are evaluated downstream.Source: §3.2 · pp. 4–5

4 EXPERIMENTS

The experiments assess masked video features for human similarity alignment, generated-video OOD detection, and retrieval, then test the masks’ design and preservation of visual information. Results compare masked features with unmasked representations and alternative objectives or feature-selection methods.

4 EXPERIMENTS

  • Experiments use V-JEPA 2.1 and VideoMAEv2 backbones, with Wan2.1 results in the main section and primary experiments repeated using CogVideoX1.5-5B and Mochi 1.The three evaluated task areas are human alignment, video-generation OOD detection, and retrieval.Source: §4 · p. 5

4.1 ALIGNMENT WITH HUMAN PERCEPTION

  • Masked features consistently improve human-similarity alignment across evaluated models; V-JEPA 2.1-L gains include +6.0 in Main Subjects ρ for the binary mask and +6.1 for the soft mask.The soft mask achieves the best overall performance, while binary masks retain 32.2% of V-JEPA 2.1-L dimensions and 42.0% of VideoMAEv2-B dimensions.Source: §4.1 · pp. 5–6, Table 1 · p. 6

4.2 VIDEO GENERATION OOD DETECTION

  • Across OOD datasets, feature spaces, and detection methods, binary masks show the highest gains, with AUROC improvements up to 34% and FPR@95 improvements up to 51%.The evaluation uses Cosmos-Predict2-generated videos and BDD100k as the in-distribution baseline.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3 VIDEO RETRIEVAL

  • Learned masks consistently improve retrieval mAP on UCF101 and HMDB51 across feature spaces, with a slight decrease for VideoMAEv2 on HMDB51.Additional generator experiments report the strongest performance from masks trained on CogVideoX.Source: Table 3 · p. 7, §4.3 · p. 7

4.4 ABLATIONS

  • 70.90% to 45.60% FPR@95: in V-JEPA DDPM-OOD, the learned mask beats the unmasked baseline, and other feature-selection or dimensionality-reduction methods do not match it across models and OOD methods.The comparison includes methods evaluated at equal sparsity; the standard similarity objective also falls significantly short of the proposed method.Source: §4.4 · p. 8, Table 4 · p. 8

4.4.1 PRESERVATION OF VISUAL INFORMATION

  • Highly sparse masked patch features retain core visual structure, while patch-clustering reconstructions preserve much of the original visual and structural information across cluster counts.The qualitative comparison uses PCA visualizations; the quantitative evaluation measures reconstruction MSE across varying cluster counts.Source: §4.4.1 · pp. 8–9, Figure 5 · p. 9, Figure 4 · p. 9

5 CONCLUSION

The paper uses text-to-video generative uncertainty to learn sparse masks over pretrained video representations for semantic video similarity. The masked representations improve results across multiple tasks and backbones while reducing dimensions, and the experiments are reproduced with three text-to-video models.

5 CONCLUSION

  • The framework uses consistency across generated videos as supervision, with general concepts tending to yield stable generations and specialized concepts higher generative divergence.It learns sparse masks over pretrained video representations without human similarity annotations or predefined similarity metrics.Source: §5 · p. 9
  • Across multiple backbones and tasks, masked representations improve alignment with human judgments, enhance OOD detection for generated videos, and maintain or improve video retrieval.The learned masks discard a large fraction of the original embedding dimensions, producing more compact representations.Source: §5 · p. 9
  • Experiments reproduced across three text-to-video models show that global masks preserve meaningful local structure and enable patch-level matching with minimal loss in segmentation quality.The paper concludes that generative-model uncertainty can serve as a signal for learning semantically useful subspaces.Source: §5 · p. 9

DISCLOSURE OF AI ASSISTANCE

The disclosure describes generative AI assistance in language polishing and experimental code implementation, with authors reviewing and verifying the resulting materials. It also identifies the tool used to generate specialized prompts and the video models used for datasets.

DISCLOSURE OF AI ASSISTANCE

  • Generative AI tools assisted with language polishing and implementation of code used in the reported experiments.The authors reviewed and verified the resulting text, code, and experimental outputs.Source: DISCLOSURE OF AI ASSISTANCE · p. 10
  • Google Gemini 3.1 Pro generated the specialized text prompts used to construct the synthetic training dataset.The disclosure points to Appendix B.1 for the dataset-construction description.Source: DISCLOSURE OF AI ASSISTANCE · p. 10
  • Wan2.1, CogVideoX1.5-5B, and Mochi 1 generated the video datasets for the proposed method.The disclosure states generative AI tools were not used for other tasks requiring disclosure.Source: DISCLOSURE OF AI ASSISTANCE · p. 10

A THEORETICAL ANALYSIS OF GENERATIVE VARIABILITY

Repeated-generation variability can supervise feature selection because it tracks how consistently a prompt yields the same semantic content. The specialized-prompt set was constructed to differ from general prompts in domain rather than superficial linguistic structure.

A.1 WHY IS GENERATIVE VARIABILITY A USEFUL LEARNING SIGNAL?

  • The analysis uses variability conditioned on repeated generations of the same prompt, without assuming specialized prompts are unseen or all variability is epistemic uncertainty.It proposes that dimensions stable when semantics remain consistent, but variable when generated content is less consistent, can identify useful features.Source: §A.1 · p. 13
  • For two independent generations from one prompt, pairwise semantic disagreement measures generator variability, linking higher variability to lower expected ideal semantic similarity.When probability mass concentrates on one semantic outcome, variability is small; when it spreads across outcomes, variability increases.Source: §A.1 · pp. 13–14

B.1 Specialized PROMPT GENERATION

  • Specialized prompts were designed to differ from general prompts in semantic domain while matching their structural distribution, reducing superficial linguistic differences as an explanation for divergence.Gemini 3.1 Pro generated 100 specialized concepts, with prompts focused on clinical surgery and cellular microscopy.Source: §B.1 · p. 14

B.2 LEARNABLE MASK TRAINING

The appendix describes mask-training settings, tests whether generative consistency transfers across video generators, and compares the learned mask with feature-selection baselines. Specialized prompts show greater diversity across generators, while attention-derived masks do not match the learned objective’s improvements.

B.2 LEARNABLE MASK TRAINING

  • The learned soft mask was trained on an 80/20 synthetic-pair split using AdamW, learning rate 1 × 10−3, batch size 128, and validation-based early stopping.Features were pre-extracted, and training took about one minute on one NVIDIA H100 GPU.Source: §B.2 · p. 15

B.3 COSMOS-PREDICT2 FINETUNING

  • The Cosmos-Predict2 2B model was fine-tuned on BDD100k for 5,000 iterations before generating the image-to-video samples used in the experiment.Fine-tuning used batch size 32, learning rate 1.1 × 10−6, and 16 NVIDIA H100 GPUs for approximately 12 hours.Source: §B.3 · p. 15

C.1 ADDITIONAL VIDEO GENERATORS

  • Across CogVideoX1.5-5B and Mochi 1, specialized prompts have higher intra-prompt feature-space diversity than general prompts in both V-JEPA 2.1-L and VideoMAEv2-B.The figure measures average pairwise normalized cosine dissimilarity among three videos generated with different seeds; qualitative examples show greater variation across three generators.Source: §C.1 · p. 15, Figure C.1 · p. 16, Figure C.2 · p. 17

C.2 FEATURE SELECTION AND DIMENSIONALITY REDUCTION BASELINES

  • Feature-selection baselines were matched to the learned binary mask’s retained dimensions: 329/1024 for V-JEPA 2.1-L and 322/768 for VideoMAEv2-B.The baselines included random, variance, PCA loading, Laplacian score, correlation filtering, and an encoder self-attention mask.Source: §C.2 · pp. 16–17
  • The self-attention-derived mask does not provide improvements comparable to the generative-consistency objective.Its channel scores combine final-block token attention with corresponding patch features, averaged over generated videos.Source: §C.2 · pp. 17–18

C.3 FULL RESULTS ON CONVIS-BENCH

On ConViS-Bench, learned masks improve human-judgment alignment over original pretrained features and generic feature selectors, with consistent gains across five semantic dimensions. The gains persist while retaining a reduced feature subspace.

C.3 FULL RESULTS ON CONVIS-BENCH

  • For V-JEPA 2.1-L, all listed generic selectors score below the unmasked representation; the strongest non-learned baseline reaches 29.74 average Spearman correlation versus 31.14 unmasked.The comparison covers random, variance, PCA loading, Laplacian score, correlation filtering, and self-attention selection.Source: §C.3 · p. 18
  • The binary mask increases average Spearman correlation to 35.44 and average Kendall correlation from 21.53 to 24.49, consistently improving all five semantic dimensions while discarding 67.8% of original feature dimensions.The soft mask reaches average Spearman correlation 36.04.Source: §C.3 · p. 18
  • For VideoMAEv2-B, the learned masks provide the strongest overall alignment, although some generic selectors improve on the unmasked representation.A random mask raises average Spearman correlation from 44.90 to 45.42; the passage reports smaller gains overall than for V-JEPA.Source: §C.3 · p. 18

C.4 DOWNSTREAM ROBUSTNESS TO THE VIDEO GENERATOR

Masks trained from CogVideoX1.5-5B and Mochi 1 transfer to semantic-alignment, OOD-detection, and retrieval evaluations, supporting robustness to the source video generator. The OOD tests also use a different generative task and model family from mask training.

C.4 DOWNSTREAM ROBUSTNESS TO THE VIDEO GENERATOR

  • For both encoders, CogVideoX1.5-5B- and Mochi 1-derived masks improve average human-similarity correlation over unmasked features on ConViS-Bench.For V-JEPA 2.1-L, binary-mask Spearman correlation rises from 31.14 unmasked to 37.20 with CogVideoX1.5-5B and 35.66 with Mochi 1; soft masks reach 38.14 and 36.78, respectively.Source: §C.4 · p. 18, Table C.2 · p. 19
  • CogVideoX1.5-5B-derived masks improve OOD detection over unmasked representations in most settings across four datasets, two encoders, and two scoring procedures.The downstream evaluation uses Cosmos-Predict2 image-to-video generation, transferring across generative tasks and model families; improvements are particularly large for MedVidBench and HAM10000.Source: §C.4 · pp. 18–19, Table C.3 · p. 20
  • Mochi 1-derived masks improve OOD discrimination in the majority of settings, and CogVideoX-derived masks yield +3.6 and +2.3 mAP on UCF101 and HMDB51 for V-JEPA features.OOD gains vary with source generator and downstream configuration; retrieval performance is reported in Table C.4.Source: §C.4 · pp. 19–20, Table C.4 · p. 20
  • Masks learned from Wan2.1, CogVideoX1.5-5B, and Mochi 1 transfer to downstream semantic-similarity and OOD-detection tasks despite differences in generator architecture, training data, and objective.Only the source of synthetic mask-training data changes; downstream datasets, feature encoders, and evaluation procedures remain unchanged.Source: §C.4 · pp. 18, 20

C.5 EXTENDING GLOBAL MASKS TO DENSE LOCAL TASKS

The learned global mask transfers to dense patch-level tasks, retaining competitive segmentation while accelerating label propagation, and masked patch features preserve visual structure in reconstructions. The broader-impact discussion highlights compactness and efficiency alongside inherited model biases.

C.5 EXTENDING GLOBAL MASKS TO DENSE LOCAL TASKS

  • The masked subspace maintains competitive DAVIS-2017 segmentation performance while accelerating label propagation by approximately 4×.The evaluation uses non-parametric patch-feature matching with the reported V-JEPA 2.1 hyperparameters.Source: §C.5 · p. 21, Figure C.3 · p. 21

C.6 FULL DETAILS ON RECONSTRUCTION VIA PATCH CLUSTERING

  • Patch tokens from masked V-JEPA 2.1 features are clustered with K-means, and cluster labels are mapped back to image locations to form a segmentation.Each cluster is rendered using the mean RGB color of its assigned pixels to create a mosaic reconstruction.Source: §C.6 · p. 22
  • Masked features’ average reconstruction MSE remains close to the full-feature baseline across evaluated k values and binarization thresholds.Figure C.4 also shows masked and unmasked reconstructions with closely matching spatial and color regions.Source: §C.6 · p. 22, Figure C.4 · p. 22

C.7 BROADER IMPACTS

  • The method is described as model-agnostic and more efficient than baseline methods, reducing representation dimensions and storage while improving alignment with human judgment across evaluated tasks.The authors note that synthesis and encoding models can pass their limitations and biases into the learned masks.Source: §C.7 · p. 22

D LIMITATIONS

The framework inherits performance and biases from its pretrained components, and its emphasis on global semantic consistency trades off marginally with fine-grained local precision. The asset list names licenses for models and datasets used in the paper.

D LIMITATIONS

  • The pipeline depends on pretrained prompt-synthesis, video-generation, and encoding models, whose biases or domain-specific shortcomings may propagate into learned masks.The authors cite potential cultural biases in text-to-video synthesis as an example.Source: D LIMITATIONS · p. 22
  • Prioritizing global semantic consistency yields compact representations but can bring a marginal performance trade-off on dense, fine-grained tasks such as video object segmentation.The authors suggest multi-scale masking as future work to bridge this gap.Source: D LIMITATIONS · p. 22

E LICENSES

  • The listed assets carry different licenses, including Apache 2.0 for Wan2.1 and Mochi 1, and MIT for V-JEPA 2.1.Source: E LICENSES · p. 23

Related papers

Loading 2609.35341v1…