PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoSSR: Video Self-Supervised Reinforcement Learning

Zefeng He, Xiaoye Qu, Yafu Li, Siyuan Huang, Daizong Liu, Yu Cheng

arXiv:2511.06281v1cs.CV

TL;DR

Existing video datasets may not adequately challenge increasingly capable MLLMs, while manual annotation of high-quality data is prohibitively expensive. VideoSSR builds self-supervised tasks and a training dataset from intrinsic video signals, and improves results across 17 benchmarks by an average of over 5%.

  • Problem

    Existing video datasets can be too easy or yield unreliable all-incorrect outcomes for powerful MLLMs, while manual annotation of new high-quality data is prohibitively expensive.

    Source: §1 · pp. 1–2

  • Method

    VideoSSR creates Anomaly Grounding, Object Counting, and Temporal Jigsaw tasks, builds the annotation-independent VideoSSR-30K dataset, and trains with GRPO and task-specific smooth rewards.

    Source: §1 · p. 2

  • Results

    Over 5% average improvement across 17 benchmarks spanning four video task domains and three input-frame settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support self-supervision as a scalable, low-cost way to generate video training data with parametrically controlled task difficulty.

    Source: §5 · p. 9

    Experiments primarily use few input frames, which may limit scalability to long videos; scaling to higher frame rates and longer inputs remains future work.

    Source: C. Limitations and Future Work · p. 14

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has substantially advanced the video understanding capabilities of Multimodal Large Language Models (MLLMs). However, the rapid progress of MLLMs is outpacing the complexity of existing video datasets, while the manual annotation of new, high-quality data remains prohibitively expensive. This work investigates a pivotal question: Can the rich, intrinsic information within videos be harnessed to self-generate high-quality, verifiable training data? To investigate this, we introduce three self-supervised pretext tasks: Anomaly Grounding, Object Counting, and Temporal Jigsaw. We construct the Video Intrinsic Understanding Benchmark (VIUBench) to validate their difficulty, revealing that current state-of-the-art MLLMs struggle significantly on these tasks. Building upon these pretext tasks, we develop the VideoSSR-30K dataset and propose VideoSSR, a novel video self-supervised reinforcement learning framework for RLVR. Extensive experiments across 17 benchmarks, spanning four major video domains (General Video QA, Long Video QA, Temporal Grounding, and Complex Reasoning), demonstrate that VideoSSR consistently enhances model performance, yielding an average improvement of over 5\%. These results establish VideoSSR as a potent foundational framework for developing more advanced video understanding in MLLMs. The code is available at https://github.com/lcqysl/VideoSSR.

1. Introduction

VideoSSR addresses limitations in existing video training data with self-supervised tasks that generate verifiable questions from videos. It achieves consistent gains across 17 benchmarks, averaging over 5%.

1. Introduction

  • Existing datasets can be too easy for powerful MLLMs or yield all-incorrect responses, producing near-zero-variance rewards and marginal gains or degradation.The authors attribute the all-incorrect cases to either intractable difficulty or biased ground truths from multi-agent annotation.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • VideoSSR uses Anomaly Grounding, Object Counting, and Temporal Jigsaw to generate verifiable questions with parametrically scalable difficulty, without human or model-generated annotations.VIUBench tests these tasks and finds that even leading closed-source models such as GPT-5 struggle with intrinsic video properties.Source: §1 · p. 2
  • The VideoSSR-30K dataset trains the VideoSSR framework with GRPO, using task-specific smooth rewards to address sparse feedback from difficult pretext tasks.The dataset is constructed from the three pretext tasks and is independent of human or MLLM annotations.Source: §1 · p. 2
  • Over 5% average gain across 17 benchmarks shows VideoSSR consistently improves performance across four video task domains and three input-frame settings.The reported experiments cover General Video QA, Long Video QA, Temporal Grounding, and Complex Reasoning.Source: §1 · p. 2

2. Related Works

Prior video RLVR methods draw on several data sources, but often rely on external annotations or target a single capability. Video self-supervision has used jigsaw tasks, and this work broadens that approach with a richer suite of pretext tasks.

2.1. Reinforcement Learning for MLLMs

  • Video RLVR methods use QA, timestamp, 3D-scene, and multi-agent data sources, but external annotations can introduce bias and single-capability training can limit broader generalization.The cited examples include Video-R1, Time-R1, SpaceR, and ReWatch-R1.Source: §2.1 · pp. 2–3

2.2. Self-supervised learning for Video

  • Video self-supervised learning uses pretext tasks to learn spatio-temporal representations from unlabeled video; prior MLLM work used jigsaw puzzles, while this paper explores a richer task suite.The figure illustrates Anomaly Grounding, Object Counting, and Temporal Jigsaw.Source: §2.2 · p. 3, Figure 3 · p. 3

3. Method

The method generates verifiable question-answer pairs directly from raw videos using three pretext tasks with parametrically controlled difficulty. It uses these tasks to construct VIUBench and VideoSSR-30K for reinforcement-learning training.

3.1. Pretext Tasks

  • The three pretext tasks generate verifiable question-answer pairs from raw videos without human- or model-generated annotations, with difficulty controlled parametrically.The tasks are Anomaly Grounding, Object Counting, and Temporal Jigsaw.Source: §3.1 · p. 3

3.1.1. Anomaly Grounding

  • Anomaly Grounding perturbs a randomly selected video interval and asks the model to identify its start and end timestamps; mIoU supplies a smooth reward based on interval overlap.Perturbations target fine-grained, spatial, or temporal perception.Source: §3.1.1 · pp. 3–4, §3.3 · p. 6

3.1.2. Object Counting

  • Object Counting inserts procedural geometric shapes into selected frames, asks for counts by shape class, and rewards predictions using averaged relative-error scores.The task’s ground truth is a vector of total counts, one for each shape class.Source: §3.1.2 · p. 4, §3.3 · p. 6

3.1.3. Temporal Jigsaw

  • Temporal Jigsaw shuffles contiguous video segments and asks the model to restore their original order; its displacement error is normalized by the maximum error for a reversed sequence.The target sequence is the inverse of the shuffle permutation.Source: §3.1.3 · p. 4, §3.3 · p. 6

3.2. Video Intrinsic Understanding Benchmark

  • VIUBench contains 2,700 question-answer pairs and evaluates Fine-grained, Spatial, and Temporal Perception across the three pretext tasks.Figure 4 shows the proportional task and subtype distributions for VIUBench and VideoSSR-30K.Source: §3.2 · p. 4, Figure 4 · p. 4
  • GPT-5 scores 58.7 on average and Qwen3-VL-8B scores 19.5 on VIUBench, showing that intrinsic video understanding remains challenging.Table 1 compares models on three abilities through Object Counting, Anomaly Grounding, and Temporal Jigsaw.Source: §3.2 · p. 5, Table 1 · p. 5
  • GPT-5’s Object Counting score drops from 88.4 to 70.3 from Easy to Hard, while Temporal Jigsaw scores fall from 39.0 to 27.0 as segments increase from six to eight.Video Jigsaw [56] also drops from 20.3 to 5.0 under the same segment change.Source: §3.2 · p. 5

3.3. Video Self-Supervised Reinforcement Learning

  • VideoSSR-30K comprises the three pretext tasks and is used to train VideoSSR with GRPO; task-specific smooth rewards address the sparse signals from strict correctness-based rewards.GRPO variants are not used because the stated focus is the data itself.Source: §3.3 · p. 5

4. Experiments

VideoSSR improves results across diverse video benchmarks, with its strongest reported average gain at the 48-frame setting. Experiments also examine the contributions of individual pretext tasks, reward design, task diversity, and training-data choice.

4. Experiments

  • Experiments cover 16 benchmarks across four task categories, with Qwen3-VL-8B-Instruct as the primary baseline and GPT-4o and Gemini-1.5-Pro as additional comparisons.Temporal Grounding uses mIoU as its primary evaluation metric.Source: §4 · pp. 6–7

4.1. Main Results

  • On Temporal Grounding, VideoSSR gains +15.9 on QVHighlights and +5.6 on ActivityNet; on Complex Reasoning, it gains +9.0 on VCRBench and improves other reasoning benchmarks.The VCRBench benchmark is described as highly correlated with the Temporal Jigsaw task.Source: Table 3 · p. 7, §4.1 · p. 7
  • VideoSSR scores 68.9 on MVBench, outperforming closed-source models, and improves four Long Video QA benchmarks while retaining a gap to closed-source models there.The authors attribute the remaining long-video gap to primarily training and evaluating with few frames.Source: §4.1 · p. 7
  • At 48 frames, VideoSSR improves average performance by 5.1% across 17 benchmarks, including VIUBench.The paper reports consistent gains across four video-task categories and three frame settings.Source: §4.1 · p. 7

4.2. Ablation Study

  • Each of the three pretext tasks individually improves Video-MME; Anomaly Grounding benefits CharadesSTA, while Temporal Jigsaw boosts VCRBench.The ablation also evaluates the three tasks jointly and separately.Source: §4.2 · p. 7, Table 4 · p. 8
  • Strict matching rewards perform closer to baseline than smooth rewards, with sparse signals causing zero GRPO advantages, inefficient updates, and training instability.Strict-reward Anomaly Grounding even degrades CharadesSTA performance.Source: §4.2 · p. 7, Table 4 · p. 8
  • At 30k samples, mixed-task training is more effective than scaling a single task, which can yield diminishing returns or degrade performance.The comparison controls for dataset scale across the two training settings.Source: §4.2 · pp. 7–8, Figure 5 · p. 8
  • VideoSSR-30K surpasses comparable-scale annotated training datasets, while LongVideoReason annotations from a less capable MLLM can degrade Qwen3-VL performance.The comparisons use multiple-choice subsets of LongVideoReason and ReWatch.Source: Table 5 · p. 8, §4.2 · p. 8

5. Conclusion

The paper introduces VideoSSR, a self-supervised reinforcement learning framework built around three pretext tasks and annotation-independent data. Experiments report consistent gains across 17 benchmarks, averaging over 5%.

5. Conclusion

  • VideoSSR is a self-supervised reinforcement-learning framework that introduces Anomaly Grounding, Object Counting, and Temporal Jigsaw to construct the challenging VIUBench and VideoSSR-30K dataset without manual or MLLM annotations.Source: §5 · p. 8
  • Across four main video tasks, the work achieves an average improvement of over 5%, and presents self-supervision as a way to generate scalable, low-cost, high-quality training data.Source: §5 · p. 9

Supplementary Materials

The supplementary material specifies the training setup and evaluation prompts. Video QA uses direct-answer prompting, while Temporal Grounding uses a unified format across benchmarks.

A.1. Training Details

  • For VideoSSR training, the authors omit chain of thought, prioritizing fine-grained, spatial, and temporal perception over complex reasoning; they report greater training efficiency and reduced potential for hallucination.Source: §A.1 · p. 12

A.2.1. Prompts

  • Video QA prompts request direct answers, while Temporal Grounding uses a unified prompt based on lmms eval; specialized benchmarks such as VCRBench use official prompts.The authors note CharadesSTA's sensitivity to Temporal Grounding prompt phrasing and retain the unified prompt for consistent evaluation.Source: §A.2.1 · p. 12, Figure 7 · p. 12, Figure 8 · p. 12

A.2.2. Benchmarks

The benchmark appendix records dataset-specific evaluation choices, including subsets, splits, and output formats.

A.2.2. Benchmarks

  • Evaluation uses text scores for VinoGround, no subtitles for Video-MME and LongVideoBench, LongVideoBench's validation set, and a 3k CGBench subset.Temporal Grounding predicts one likely interval; VideoMMMU and Video-TT use multiple-choice subsets, while CVBench tests 32, 48, and 64 frames.Source: §A.2.2 · p. 12

A.2.3. Detailed Results

The detailed results section points to additional Temporal Grounding results, while VIUBench evaluation uses a fixed 48-frame input at a maximum resolution of 512 × 512 pixels.

A.2.3. Detailed Results

  • Tables 6 and 7 provide further Temporal Grounding results for QVHighlights, ActivityNet, CharadesSTA, and Tacos.The passages identify the tables’ scope but do not provide their result values.Source: §A.2.3 · p. 12

A.3. Evaluation Details for VIUBench

  • VIUBench evaluations use 48 frames at a maximum resolution of 512 × 512 pixels.Source: §A.3 · p. 12

B.1. Anomaly Grounding

The appendix details the Anomaly Grounding, Object Counting, and Temporal Jigsaw prompts and examples. For Slow/Fast anomaly tasks, models are directed to use visual motion rather than timestamps, yet performance remains poor.

B.1. Anomaly Grounding

  • Anomaly Grounding uses 14 perturbation subtypes, with prompt descriptions supplied for each subtype and extra guidance for temporal-perception tasks.The prompt template is shown in Figure 9.Source: §B.1 · p. 12, Figure 9 · p. 13
  • Slow/Fast anomaly prompts instruct models to ignore evenly spaced timestamps and judge playback speed from visual motion, but performance remains poor.The authors hypothesize that Qwen3-VL has a strong bias toward textual timestamp information.Source: §B.1 · pp. 12–13
  • Tables 6 and 7 give additional Temporal Grounding results for QVHighlights, ActivityNet, CharadesSTA, and Tacos.The supplied captions specify the datasets but not the result values.Source: Table 6 · p. 13, Table 7 · p. 13
  • The model predicts the temporal range of an introduced anomaly from visual evidence; Figures 12–15 show examples across four perturbation types.Only a subset of key frames is displayed in the examples.Source: §B.1 · p. 13

B.2. Object Counting

  • Object Counting uses a prompt template and examples whose ground-truth circle, square, and triangle counts are shown in Figures 16 and 17.Source: §B.2 · p. 13, Figure 10 · p. 13

B.3. Temporal Jigsaw

  • Temporal Jigsaw presents a shuffled clip sequence for ordering, alongside the corresponding original video for comparison.The prompt template and an example shuffled sequence are shown in Figures 11 and 18; Figure 19 shows the original video.Source: §B.3 · p. 13

B.4. Exploration of Alternative Pretext Tasks

The authors explored generative reconstruction, next-frame prediction, and temporal speed prediction as alternative self-supervised tasks. Preliminary experiments found no significant or consistent downstream gains from these alternatives.

B.4. Exploration of Alternative Pretext Tasks

  • The explored generative tasks were masked-frame reconstruction and autoregressive next-frame prediction.Source: §B.4 · p. 13
  • The authors also tested direct temporal speed prediction as an alternative pretext task.Source: §B.4 · p. 13
  • Preliminary experiments found no significant or consistent downstream benchmark improvements from these alternatives, and broader effective self-supervised tasks remain a future direction.The authors suggest these objectives may align less directly with the targeted perceptual and reasoning skills.Source: §B.4 · pp. 13–14

C. Limitations and Future Work

The authors identify low frame counts and reliance on only three pretext tasks as limitations. They propose scaling video inputs and broadening the task suite and mixing strategies as future work.

C. Limitations and Future Work

  • Experiments primarily use few input frames for efficiency, which may limit scalability to long videos; higher frame rates and longer inputs are proposed for future work.The authors note this could matter for complex, long-form content with dense temporal information.Source: C. Limitations and Future Work · p. 14
  • The framework relies on three pretext tasks, leaving broader objectives and possible task-mixing strategies for future exploration.Suggested strategies include curriculum learning and adaptive task weighting.Source: C. Limitations and Future Work · pp. 14–15

Related papers

Loading 2511.06281v1…