PaperlayerView exact revision on arXiv

Source-linked AI summary

Find the Leak, Fix the Split: Cluster-Based Method to Prevent Leakage in Video-Derived Datasets

Noam Glazner, Noam Tsfaty, Sharon Shalev, Avishai Weizman

arXiv:2511.13944v2cs.CV

TL;DR

Video-derived frames can be so spatially and temporally correlated that random splitting risks information leakage and inflated evaluation estimates. The paper groups visually related frames before partitioning, and reports that DINO-V3 embeddings achieve the best clustering across ImageNet-VID and UCF101.

  • Problem

    Randomly splitting video-derived images can leak information across partitions because consecutive frames are highly correlated, inflating performance estimates.

    Source: §I · p. 1

  • Method

    The method extracts frame features, projects them with PaCMAP, clusters the embeddings with HDBSCAN, and assigns each cluster to one dataset partition.

    Source: A. Feature Extraction · p. 1, B. Dimensionality Reduction and… · p. 2, C. Cluster-Based Dataset Partitioning · p. 2

  • Results

    DINO-V3 embeddings achieve the best clustering across ImageNet-VID and UCF101, while deep pretrained representations outperform classical and lightweight descriptors on both datasets.

    Source: §IV · p. 2, §III · p. 2

  • Takeaways & Limitations

    The authors conclude that grouping visually similar and temporally correlated frames before partitioning can improve dataset diversity and provide a fairer evaluation setting.

    Source: §IV · p. 2

    The approach relies on HDBSCAN and its hyperparameter choices.

    Source: §IV · p. 2

Abstract

from arXiv · show

We propose a cluster-based frame selection strategy to mitigate information leakage in video-derived frames datasets. By grouping visually similar frames before splitting into training, validation, and test sets, the method produces more representative, balanced, and reliable dataset partitions.

I. INTRODUCTION

Randomly splitting images from videos can leak information because consecutive frames are highly correlated, potentially inflating performance estimates. The paper proposes grouping visually similar frames before partitioning to keep correlated images together.

I. INTRODUCTION

  • Video-derived frames can be highly spatially and temporally correlated, so random image-level splitting can place near-duplicate frames across partitions.The frames may share a background and objects while differing only slightly in position or rotation.Source: §I · p. 1
  • Such leakage can inflate performance estimates, particularly for image-based object detection trained on video-derived frames.Source: §I · p. 1
  • The proposed strategy groups visually similar frames before splitting so correlated images remain together in train, validation, or test partitions.The authors describe it as simple, scalable, and integrable into existing dataset-preparation pipelines without changing training.Source: §I · p. 1
  • The authors state that this partitioning approach can improve fairness and robustness of model evaluation on video-derived frames.Source: §I · p. 1

II. PROPOSED METHOD

The proposed framework is illustrated in Figure 1.

II. PROPOSED METHOD

  • Figure 1 presents an overview of the proposed framework.Source: §II · p. 1

ORCID: 0009-0004-1182-8601

The framework begins with a collection of unlabeled videos, each decomposed into extracted frames. The accompanying illustration places feature extraction, dimensionality reduction, and clustering before dataset partitioning.

ORCID: 0009-0004-1182-8601

  • The illustrated pipeline extracts frame features, reduces dimensionality, and clusters visually similar frames before partitioning the dataset.Source: Figure 1 · p. 1
  • The input is a collection of K unlabeled videos, with each video V_k decomposed into its sequence of N_k frames.Source: ORCID: 0009-0004-1182-8601 · p. 1

A. Feature Extraction

Feature extraction represents each frame using handcrafted or learned descriptors, ranging from local texture and shape features to semantic image representations. VLAD can aggregate local descriptors into a compact fixed-length vector.

A. Feature Extraction

  • SIFT and HOG are described as handcrafted descriptors that encode local gradient-orientation patterns to capture texture and shape.Source: A. Feature Extraction · p. 1
  • XFeat provides lightweight keypoint detection and description, while pretrained CLIP, SigLIP, and DINO-V3 models provide semantic image representations.Source: A. Feature Extraction · p. 1
  • VLAD assigns local descriptors to nearest codewords and aggregates their residuals into a compact, fixed-length representation.Source: A. Feature Extraction · p. 2

B. Dimensionality Reduction and Clustering

Frame features are projected into a low-dimensional space with PaCMAP, then clustered with HDBSCAN. The paper favors density-based clustering because it accommodates irregular cluster shapes and variable frame densities.

B. Dimensionality Reduction and Clustering

C. Cluster-Based Dataset Partitioning

HDBSCAN groups frames by similarity in their feature representations. Each resulting cluster contains visually related frames, and the method assigns each cluster to a single dataset partition to prevent leakage between sets.

C. Cluster-Based Dataset Partitioning

III. EXPERIMENTS AND RESULTS

Across ImageNet-VID and UCF101, DINO-V3 achieves the highest reported AMI and V-measure among the evaluated feature methods. The experiments compare HDBSCAN clustering against ground-truth videos using dataset-specific frame and feature settings.

III. EXPERIMENTS AND RESULTS

  • The study evaluates ImageNet-VID validation data and all UCF101 partitions, sampling one frame per second from UCF101 videos.The sampling was used to reduce visual redundancy and avoid nearly identical consecutive frames.Source: §III · p. 2
  • Images were resized to 224×224 for XFeat, CLIP, DINO, and SigLIP; HOG used 128×128, which performed slightly better than its 224×224 setting.VLAD vectors were reduced to 1024 dimensions, and PaCMAP embeddings to 256 dimensions.Source: §III · p. 2
  • DINO-V3 achieves the highest V-measure and AMI scores on both ImageNet-VID and UCF101.The table reports HDBSCAN clustering performance for different feature extractions on both datasets using PaCMAP (256-D).Source: Table I · p. 2, §III · p. 2
  • Clustering quality is assessed with AMI and V-measure, which measure correspondence between clusters and ground-truth videos.Source: §III · p. 2
  • Deep pretrained representations outperform classical and lightweight descriptors on both datasets, while XFeat + VLAD improves over traditional descriptors.SIFT + VLAD is weakest on both datasets, especially UCF101.Source: §III · p. 2

IV. CONCLUSIONS AND FUTURE WORK

The paper presents cluster-based frame selection to group visually similar, temporally correlated frames before partitioning video datasets. It reports DINO-V3 as the strongest clustering representation across the two evaluated datasets, while identifying HDBSCAN hyperparameter reliance as a limitation.

IV. CONCLUSIONS AND FUTURE WORK

  • The proposed strategy groups visually similar and temporally correlated frames before dataset partitioning to mitigate leakage.The authors state that this can improve diversity, provide a fair evaluation setting, and mitigate overfitting in object detection models trained on video datasets.Source: §IV · p. 2
  • DINO-V3 embeddings achieve the best clustering across ImageNet-VID and UCF101.Source: §IV · p. 2
  • The approach relies on HDBSCAN and its hyperparameter choices, a limitation the authors plan to address with adaptive clustering strategies.Source: §IV · p. 2
  • The authors plan to measure the effect of information leakage by training image-based object detection models with and without the approach across datasets.Source: §IV · p. 2
Loading 2511.13944v2…