PaperlayerView exact revision on arXiv

Source-linked AI summary

Less from More: Reinforcing Sparse Video Reasoning from Dense References

Wenfang Sun, Yingjun Du, Cees G. M. Snoek

arXiv:2610.10893v1cs.CV

TL;DR

Video-language models often rely on dense temporal observations, but dense inputs are costly and sparse sampling can remove evidence needed for reliable reasoning. SAVER uses dense views as training-time references to post-train sparse-frame reasoning with grounding and reliability-gated reference rewards. Across temporal-grounding and video-QA benchmarks, it improves performance across frame budgets, with sparse SAVER nearly matching a dense Qwen3.5 baseline in a reported video-QA comparison.

  • Problem

    Dense temporal inputs incur substantial visual-token, memory, and latency costs, while sparse sampling can remove evidence needed for reliable answers or event localization.

    Source: §1 · p. 2

  • Method

    SAVER post-trains paired dense and sparse video views with grounding rewards and a reliability-gated reference reward that aligns sparse predictions with better dense-view predictions.

    Source: §1 · p. 3, §3.4 · p. 6

  • Results

    Across three temporal-grounding and six video-QA benchmarks, SAVER improves performance across frame budgets; at 0.1 fps, SAVER-2B nearly matches dense Qwen3.5-2B at 2.0 fps.

    Source: §1 · pp. 2–3, §5 · p. 10

  • Takeaways & Limitations

    SAVER’s results support using temporal grounding as a training task for sparse video reasoning that transfers to broader video-QA benchmarks.

    Source: §1 · pp. 2–3

    The reference gate does not certify dense predictions as correct or guarantee that the combined reward increases monotonically with grounding quality.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Video-language models commonly assume that more temporal observations lead to more reliable reasoning. We question this assumption and argue that the key challenge is not merely processing more video frames efficiently, but learning to reason reliably under limited temporal evidence. We propose SAVER, a dense-to-sparse post-training framework that uses dense video views as training-time references for sparse-frame inference. During reinforcement post-training, paired dense and sparse views are optimized with grounding rewards and a reliability-gated reference reward, encouraging sparse view predictions to preserve task-relevant temporal evidence. Notably, SAVER is trained only on 1,250 randomly sampled temporal grounding examples, without using any video question answering annotations. Across three temporal grounding benchmarks and six video question-answering benchmarks, SAVER consistently improves performance across frame budgets. In particular, SAVER can match or surpass dense-frame Qwen3.5 baselines while using substantially fewer frames. These results show that temporal grounding can serve as an effective evidence-localization proxy for learning sparse video reasoning that transfers to broader video understanding tasks.

1 Introduction

Dense video inputs can improve reasoning but incur substantial visual-token, memory, and latency costs, while simply reducing frames may remove evidence needed for reliable answers or event localization. SAVER uses dense views during training to improve sparse-frame reasoning, with gains reported across temporal-grounding and video-QA benchmarks.

1 Introduction

  • Dense temporal inputs can involve tens or hundreds of frames, increasing visual-token budgets, memory use, and latency, especially challenging under limited compute or latency budgets.The paper highlights streaming video, mobile assistants, robotics, surveillance, and long-form video search as settings where these costs matter.Source: §1 · p. 2
  • The central challenge is to make sparse-frame inference reliable, because reducing frame rate can remove evidence needed for answering questions or localizing events.Dense sampling can also introduce redundancy and computational cost.Source: §1 · p. 2
  • Across three temporal-grounding and six video-QA benchmarks, SAVER improves performance across frame budgets; at 0.1 fps, SAVER-2B nearly matches dense Qwen3.5-2B at 2.0 fps.The reported video-QA transfer extends beyond the temporal-grounding task used for post-training.Source: Figure 2 · p. 2, §1 · pp. 2–3
  • Prior efficiency approaches reduce language-side computation or compress, select, or retrieve visual evidence from richer inputs, whereas this work targets reliability under a prescribed low-frame-rate budget.The distinction is whether the model can reason reliably from sparse observations themselves.Source: §1 · pp. 2–3
  • SAVER uses paired dense and sparse views during post-training, applying grounding rewards and a reference reward to align sparse predictions with reliable dense predictions.Dense views serve as training-time references, not as a requirement at inference.Source: §1 · p. 3
  • SAVER is trained on 1,250 randomly sampled temporal-grounding examples without using video-QA annotations.Despite this grounding-only training set, the paper reports improvements on six video-QA benchmarks.Source: §1 · p. 3

2 Related Work

Prior visual-reasoning post-training and video-efficiency methods generally assume sufficient visual evidence at inference or reduce the cost of richer inputs. SAVER instead uses dense views as training scaffolds to improve reasoning from uniformly sampled sparse frames at inference.

Visual reasoning models

  • Visual-reasoning post-training methods improve reasoning behavior or supervision signals, but generally assume sufficient visual evidence is available at inference.This includes reinforcement post-training for image-, region-, and video-level reasoning.Source: §2 · p. 3

Efficient video understanding

  • Video-efficiency methods reduce language-side computation or compress, select, or retrieve frames from richer video contexts.Examples include token compression, adaptive sampling, and frame retrieval.Source: §2 · p. 3
  • SAVER differs by evaluating uniformly sampled sparse frames under a fixed low-frame-rate budget and using dense views only during training to align sparse predictions with dense-view signals.It does not propose a new frame selector or token compressor.Source: §2 · p. 3

3 Method

SAVER trains paired dense and sparse views so dense predictions guide sparse predictions only when their grounding reward is higher. At inference, it uses only the sparse view.

3.4 Training Objective and Inference

  • At inference, SAVER uses only sparse frames, without dense pre-scanning, adaptive frame selection, or test-time optimization.The protocol is compatible with low-fps streaming or resource-constrained settings, where only a sparse frame stream is decoded or processed.Source: §3 · p. 4, §3.4 · p. 5

3.1 Overview

  • For each training example, SAVER samples dense and sparse views, runs both through the same MLLM policy, and optimizes their responses with grounding and reference rewards.The resulting sparse-view policy is evaluated on both temporal grounding and video question answering to test cross-task transfer.Source: §3.1 · p. 4

3.2 Dense-to-Sparse Post-Training

  • SAVER uses 1,250 randomly sampled Time-R1 temporal-grounding examples and no video-QA annotations during post-training.Temporal grounding supplies explicit supervision about when an event occurs; video-QA gains therefore reflect cross-task transfer rather than task-specific QA post-training.Source: §3.2 · p. 4

3.3 Reward Design

  • Three rewards shape post-training: valid-output format, per-view temporal grounding, and dense-to-sparse reference agreement gated by higher dense grounding reward.The grounding reward combines temporal intersection-over-union with a boundary penalty; the gate is a relative, sample-level criterion, not a guarantee of dense-prediction correctness or monotonic reward improvement.Source: §3.3 · pp. 4–5

3.4 Training Objective and Inference

  • Unlike conventional same-input self-distillation, SAVER uses reward-guided cross-view distillation and aligns sparse predictions only when the dense prediction scores higher.The risk intuition links sparse-view performance to dense-prediction reliability and sparse–dense discrepancy, which the grounding and gated reference rewards target.Source: §3.4 · pp. 5–6

4 Experiments

SAVER is evaluated across temporal grounding and video-QA benchmarks under controlled frame budgets, with dense-to-sparse post-training improving sparse-frame results. The analysis examines reward choices, alternative training strategies, reward-weight sensitivity, and remaining failure cases.

4.1 Experiment Details

  • Training uses 1,250 randomly sampled Time-R1 grounding examples, while evaluation covers three grounding benchmarks and six video-QA benchmarks at frame rates from 0.1 to 2 fps.No video-QA annotations are used for post-training; test-time SAVER uses sparse frames, with sampled frames capped at 4–256 per video.Source: §4.1 · p. 6

4.2 Main Results

  • At 0.1 fps, SAVER raises average mIoU from 18.4% to 30.0% (2B) and from 19.7% to 28.2% (4B) over Qwen3.5.For the 2B model, average mIoU also rises from 34.9% to 45.4% at 2.0 fps.Source: §4.2 · pp. 6–7, Table 1 · p. 7
  • Across six video-QA benchmarks, SAVER improves macro-averaged accuracy over Qwen3.5 across evaluated scales and frame rates.At 2.0 fps, accuracy rises from 51.0% to 54.6% for 2B and from 60.8% to 64.4% for 4B; at 0.1 fps, 2B rises from 45.9% to 50.8%. Some settings regress, and reducing sampling from 2.0 to 0.1 fps lowers average observed frames from 139 to 38 (72.7%).Source: §4.2 · p. 7, Table 2 · p. 8
  • In a sparse-frame example, Qwen3.5 predicts a broader temporal interval than SAVER, whose segment is closer to the ground truth and dense-view prediction.The comparison illustrates the qualitative grounding behavior when only a few frames are available.Source: §4.2 · p. 8, Figure 3 · p. 9

4.3 Analysis

  • At 0.1 fps, the gated reward variant reaches 30.0 mIoU and 50.8% average video-QA accuracy, the highest scores among the reward ablations.Joint dense-and-sparse training reaches 27.4 mIoU and 48.7% accuracy; adding the reference reward raises these to 29.1 and 49.3%.Source: §4.3 · p. 8, Table 3 · p. 9
  • Among the evaluated training configurations, SAVER achieves the highest scores, with 30.0 mIoU and 50.8% average video-QA accuracy.The comparison includes Time-R1 RL, sparse SFT, and a ground-truth-segment reference variant; the dense-reference variant has a modestly higher QA score than the ground-truth reference.Source: §4.3 · p. 8, Table 4 · p. 9
  • Increasing the sparse-view reward weight improves grounding mIoU and video-QA accuracy, but an overly large weight weakens the balance with dense-view scaffolding; λs = 2 is the reported best trade-off.Reference-reward weight sensitivity shows an intermediate optimum rather than monotonic gains from increasing its magnitude.Source: Figure 4 · p. 9, §4.3 · p. 9
  • SAVER can still predict a broad or shifted interval when sparse frames miss the key event, and may align to the wrong occurrence of similar actions, especially at very low frame rates.Adaptive online sampling is proposed as a future direction for cases where sparse observations are insufficient.Source: §4.3 · pp. 9–10

5 Conclusion

SAVER uses dense views as training-time references to improve sparse-input video understanding. With limited temporal-grounding training and no video-QA annotations, it improves performance across frame budgets while retaining a fixed sparse-input inference protocol.

5 Conclusion

  • SAVER jointly trains dense and sparse views with grounding supervision and a relative-quality gate controlling an interval-level reference reward.Post-training uses 1,250 grounding examples and no video-QA annotations.Source: §5 · p. 10
  • SAVER improves macro-averaged performance across evaluated model scales and frame budgets, while video-QA gains vary in magnitude and direction across benchmarks.The video-QA results demonstrate cross-task transfer.Source: §5 · p. 10
  • Inference requires neither dense video pre-scanning, adaptive frame selection, nor test-time optimization, preserving a fixed sparse-input protocol.The framework learns to use limited temporal evidence through post-training rather than selecting more informative frames at inference.Source: §5 · p. 10
  • The findings support training-time dense references as a practical approach to improving performance under limited temporal observations.Source: §5 · p. 10

AI use statement

The authors describe AI assistance across research and manuscript tasks, while specifying areas where AI was not used. They state that AI-assisted work was reviewed and verified, and outline planned reproducibility materials.

AI use statement

  • AI tools assisted with conceptual framing, research hypotheses, reward and experiment discussions, coding, literature review, figures, and manuscript drafting and editing.Source: AI use statement · p. 10
  • The authors report no AI use for synthetic datasets, theoretical models, mathematical claims or proofs, translation, dataset cleaning, qualitative analysis, or interpretation of results.Source: AI use statement · p. 10

Reproducibility statement

  • The authors state that AI-assisted work was critically reviewed and verified, and that code, model checkpoints, and the sampled training subset are planned for release.They report manually testing AI-assisted code and checking figures and tables against original experimental results.Source: AI use statement · p. 10, Reproducibility statement · p. 10

A Controlled Frame-Rate Evaluation

The controlled frame-rate evaluation reports task performance alongside observed-frame counts, with a per-video frame cap. Under matched frame budgets, SAVER improves average performance and narrows the sparse–dense gap, especially at low frame rates.

A Controlled Frame-Rate Evaluation

B Effect of sparse view Training Frame Rate

The ablation fixes dense-view training at 2.0 fps, varies the sparse-view training rate, and evaluates at 0.1-fps inference. A moderate 0.5-fps sparse training view gives the strongest reported trade-off across grounding and video-QA benchmarks.

B Effect of sparse view Training Frame Rate

C Practical Efficiency Profiling

Profiling compares Qwen3.5-2B at 2.0 and 0.1 fps across the nine temporal-grounding and video-QA benchmarks. Lower frame rates reduce visual tokens and practical inference costs on average, although MVBench is an exception.

C Practical Efficiency Profiling

  • Profiling used Qwen3.5-2B on the same nine benchmarks, with batch size 1, BF16 precision, and one warm-up generation on a single 96-GB NVIDIA H100 GPU.End-to-end latency includes video loading, frame sampling, preprocessing, host-to-device transfer, and generation; GPU latency measures synchronized model.generate() time.Source: C Practical Efficiency Profiling · pp. 15–16
  • 86.8% fewer visual tokens accompanied 63.6% lower end-to-end latency (2.74× speedup), 63.9% lower GPU latency, and 49.3% lower peak memory at 0.1 versus 2.0 fps.Averaged across nine benchmarks, the practical savings extend beyond the nominal frame count.Source: Table 8 · p. 16, C Practical Efficiency Profiling · p. 16
  • On MVBench, latency increased at 0.1 fps as average output length rose from 8.8 to 43.3 tokens, with end-to-end latency rising from 0.59 to 1.31 seconds and GPU latency from 0.49 to 1.22 seconds.The authors associate this exception primarily with longer output generation rather than increased visual processing.Source: C Practical Efficiency Profiling · p. 16

D Comparison with Inference-Time Selection Methods

SAVER improves sparse-frame reasoning through post-training, while FastVID and WFS-SB select or prune from richer visual inputs. The approaches can be combined, and SAVER has lower measured inference costs under matched profiling.

D Comparison with Inference-Time Selection Methods

  • FastVID and WFS-SB select or retain informative observations from a richer candidate pool, whereas SAVER post-trains the model to reason from a fixed sparse input budget.The paper presents these as distinct approaches that can be composed when a richer candidate pool is available.Source: D Comparison with Inference-Time… · pp. 16–17
  • At 0.1 fps, SAVER scored 51.0 on average across three benchmarks, versus 47.1 for FastVID and 49.0 for WFS-SB using the same Qwen3.5-2B backbone.Table 9 compares the approaches on one temporal-grounding benchmark and two video-QA benchmarks.Source: D Comparison with Inference-Time… · p. 16
  • Under matched profiling, SAVER used fewer visual tokens, less peak GPU memory, and lower GPU and end-to-end latency than FastVID and WFS-SB.SAVER operates directly on sparse inputs without dense pre-scanning, intermediate feature extraction, or an additional selection stage at inference.Source: Table 10 · p. 17, D Comparison with Inference-Time… · p. 17
  • Combining SAVER with WFS-SB increased average performance from 51.0 to 55.0, including ActivityNet from 37.2 to 42.1 and LongVideoBench from 56.0 to 62.3.The paper describes training-time sparse robustness and inference-time adaptive selection as complementary.Source: D Comparison with Inference-Time… · p. 17

E Effect of Reference Reward Weight

The section examines reference-reward weighting, grounding-data scale, and an optional sparse-to-dense fallback. Results show trade-offs in reward balance and fallback calibration, while grounding-only transfer saturates with a relatively small training set.

E Effect of Reference Reward Weight

  • Increasing λref to 1.0 improved temporal-grounding and video-QA performance, but raising it to 2.0 changed grounding only from 30.0 to 30.1 mIoU and reduced QA accuracy from 50.8 to 50.2.The authors attribute the benefit to a distinct cross-view alignment objective balanced with other reward components, not simply a larger reference reward.Source: E Effect of Reference Reward Weight · p. 17

F Effect of Grounding Post-Training Data Scale

  • Grounding supervision initially improved both tasks, but gains saturated beyond 1,250 examples; using all 2,500 Time-R1 examples slightly degraded sparse-inference performance.The authors hypothesize that excessive grounding-focused post-training may over-specialize the model toward narrow localization patterns.Source: Table 12 · p. 18, F Effect of Grounding Post-Training… · p. 18

G Confidence-Based Sparse-to-Dense Fallback

  • At γ = 0.7, fallback raised grounding from 30.0 to 38.3 average mIoU and video-QA from 50.8 to 52.0 average accuracy, using 47 versus 110 and 54 versus 139 frames, respectively.Lower thresholds can misroute correct sparse predictions because confidence is not sufficiently calibrated; the paper treats fallback as optional.Source: Table 13 · p. 18, G Confidence-Based Sparse-to-Dense… · p. 19

H Additional Qualitative Cases

The qualitative examples show successful sparse reasoning when sampled frames retain useful cues, alongside failures when actions are ambiguous, key evidence is missed, or an incorrect premise guides the reasoning.

H Additional Qualitative Cases

  • For “a person opens a small cabinet door,” SAVER predicted an interval closely matching the ground truth, from the clip’s beginning to around 4 seconds.The example illustrates success when sparse frames preserve the key temporal evidence.Source: Figure 5 · p. 20
  • For “person throws their blanket inside,” SAVER predicted an earlier interval because sparse observations did not distinguish the target action from a plausible doorway movement.The example identifies ambiguous or insufficiently captured actions as a failure mode.Source: Figure 6 · p. 21
  • On a VideoMMMU mechanics question, SAVER followed the visual solution, applied the changed length to the formula, and selected the correct answer.The example’s success depended on retaining the necessary visual and textual evidence in the sampled frames.Source: Figure 7 · p. 22
  • On a VideoMMMU chemistry question, SAVER chose 32 kg using a 1:8 mass ratio, while the ground truth was 64 kg under the video’s 1:16 relation.The arithmetic was internally consistent, but the model relied on a default prior rather than the relation presented in the video.Source: Figure 8 · p. 23
Loading 2610.10893v1…