PaperlayerView exact revision on arXiv

Source-linked AI summary

Moving Forward with Video Saliency: A New Dataset and Benchmark where Motion Matters

Susmit Agrawal, Rebecca Wanner, Juliane Verwiebe, Matthias Tangemann, Matthias Bethge, Matthias Kümmerer

arXiv:2610.03276v1cs.CV

TL;DR

Static baselines recover much of LEDOV’s explainable gaze information, leaving unclear whether temporal models are limited or the benchmark lacks temporal patterns. The paper introduces SalTempto, a more dynamic video saliency benchmark, and finds a larger temporal-model advantage there than on LEDOV. Even so, the strongest temporal model leaves substantial SalTempto headroom unexplained.

  • Problem

    LEDOV results leave unclear whether modest temporal-model gains reflect architecture limits or a lack of temporal patterns in the benchmark.

    Source: §1 · p. 2

  • Method

    The paper introduces SalTempto, a benchmark of 224 dynamic HACS-Segments clips with gaze recordings from up to 16 subjects, and evaluates static and temporal models.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    SalFoM-Tuned exceeds DG-MR-Tuned by 45 percentage points on SalTempto, versus 19 percentage points on LEDOV; SalFoM leaves approximately 41.5% of SalTempto headroom unexplained.

    Source: §4 · p. 5

  • Takeaways & Limitations

    The results indicate that LEDOV does not measurably require temporal reasoning, while SalTempto provides a setting to address that mismatch.

    Source: §6 · p. 10

    SalTempto targets foreground human-activity dynamics, so the conclusions do not cover content whose dynamics come from no acting agent.

    Source: §5 · p. 9

Abstract

from arXiv · show

Video saliency prediction is inherently harder to model than static image saliency due to the additional temporal dimension. Video saliency benchmarks rest on the premise that predicting gaze on video requires utilizing temporal activity distributed across frames. Prior work has challenged this, showing that static baselines recover a significant fraction of the explainable gaze information on LEDOV, a popular video saliency dataset, and that video saliency models fail in the same places as this static baseline. We verify that this diagnosis still stands: under a more capable gold standard than the original analysis, and an updated panel of recent architectures, the strongest temporal architecture in the panel still does not substantially improve over a fine-tuned static baseline. However, it remains unclear whether the marginal gain reflects limitations of current temporal architectures or a lack of temporal patterns in the benchmark itself. We introduce SalTempto, a video saliency benchmark with greater dynamism: 224 clips of highly dynamic content, sourced from the HACS-Segments dataset so that each clip contains an event together with its lead-up and aftermath, with gaze recordings from up to 16 subjects and a training split for adapting pretrained models. On SalTempto, the static baseline recovers only about 13\% of the headroom above the centerbias, against more than half on LEDOV. A fine-tuned temporal architecture shows a substantial gain in performance over the static baseline, indicating that it does capture meaningfully more temporal information, which LEDOV fails to measure. Yet, even this SoTA model still leaves nearly half of SalTempto's headroom unexplained, indicating room for improvement in video saliency modelling. Examination of SalTempto also lets us describe human tendencies that models miss. SalTempto link: https://huggingface.co/datasets/bethgelab/video_saliency.

1 INTRODUCTION

Video saliency models should use viewing history and scene dynamics, but prior LEDOV results show that static features can recover much of the explainable gaze information. The paper confirms this pattern with recent models and introduces SalTempto, a more dynamic benchmark where temporal models show a substantial advantage over a static baseline.

1 INTRODUCTION

  • Video saliency models are intended to predict how attention flows using viewing history and scene dynamics, rather than only current-frame visual attraction.The temporal premise motivates architectures with recurrent backbones, 3D convolutions, and transformers.Source: §1 · p. 1
  • On LEDOV, the static baseline recovers 56% of headroom above centerbias, while the strongest current temporal model reaches 75%, only a 19-point improvement.The authors find gaze targets cluster spatially across time, so one frame’s gold-standard density predicts nearly 30% of another frame’s net available gaze information.Source: §1 · p. 2
  • SalTempto comprises 224 one-minute clips of highly dynamic content, each containing an annotated event with its lead-up and aftermath, and gaze recordings from up to 16 subjects.The clips are sourced from HACS-Segments; their foreground dynamics yield wider within-video gaze dispersion than LEDOV.Source: §1 · p. 2
  • On SalTempto, a fine-tuned temporal architecture substantially outperforms the static baseline, but still leaves approximately 41.5% of headroom unexplained.The authors also identify object permanence despite occlusion, scene inertia, and anticipatory saccades as viewer behaviours models tend to miss.Source: §1 · p. 2

2 RELATED WORK AND EXISTING DATASETS

LEDOV is a preferred benchmark for fine-tuning and evaluating video saliency architectures, but other available datasets have limitations for diagnosing temporal modelling. Prior work proposed a static-recoverability diagnostic, while a stand-alone dataset with greater dynamism is needed to test temporal capabilities.

2 RELATED WORK AND EXISTING DATASETS

  • DHF1K has fixation-mask artefacts and lacks per-subject gaze identity, while Hollywood-2 and UCF-Sports use task-aware viewing and each focus on one content domain.DIEM has audio playback and heterogeneous recording setups; the paper reports a zero-shot comparison on it despite those confounders.Source: §2 · p. 2
  • Tangemann et al. (2020) introduced the static-recoverability diagnostic and DG-MR, then assembled a meta-dataset of temporally demanding examples, but not a stand-alone dynamic benchmark.SalTempto provides the stand-alone benchmark with greater dynamism that the prior contribution lacked.Source: §2 · pp. 2–3
  • Figure 1 compares model information gain over centerbias and headroom recovered on LEDOV and SalTempto; its DG-MR-to-SalFoM climb is steeper on SalTempto.The bars distinguish recovered headroom from the remaining unexplained portion.Source: Figure 1 · p. 3

3 METHODS

The methods construct SalTempto from dynamic HACS-Segments clips, collect gaze at different densities across training and evaluation splits, and benchmark static and temporal models with shared information-theoretic measures. The benchmark also defines centerbias and a leave-one-subject-out gold standard for evaluating model performance and remaining headroom.

3.1 SALTEMPTO DATASET CREATION

  • The authors select 224 high-resolution HACS-Segments clips whose annotated action begins at least 15 seconds in, has midpoint at least 30 seconds, and ends before the clip ends.HACS-Segments spans 200 categories; action dynamics, camera motion, and scene cuts contribute to gaze dynamics, and the selected clips yield greater within-video gaze dispersion than LEDOV.Source: §3.1 · p. 3
  • Sixteen participants provided gaze recordings; each training video was viewed by 3 subjects, while each validation and test video was viewed by all 16.The split balances video diversity for training against denser gaze samples for validation and test analyses.Source: §3.1 · p. 3
  • The 500 Hz gaze streams are binned by video frame while preserving subject identity, and the dataset releases raw trajectories alongside binned positions.The raw trajectories allow downstream recovery of saccade and smooth-pursuit dynamics that frame averaging or fixation stabilisation would discard.Source: §3.1 · p. 4
  • SalTempto is split into 204 training, 10 validation, and 10 test videos, with public training and validation gaze data and held-out test recordings.The text reports 3.4 hours of training video and 0.17 hours each for validation and test.Source: §3.1 · p. 4

3.2 THE SALTEMPTO BENCHMARK

  • The benchmark evaluates 13 architectures zero-shot and fine-tunes DG-MR and the panel’s strongest temporal model, SalFoM, on each dataset’s training split.A control evaluates fine-tuned SalFoM with temporal context removed by repeating the target frame across its context window.Source: §3.2 · p. 4
  • Each dataset’s centerbias is a Gaussian KDE over training fixations, with bandwidth chosen by held-out log-likelihood: σ = 3.15% on SalTempto and 4.44% on LEDOV.The different optima reflect the datasets’ fixation spread, not a methodological asymmetry.Source: §3.2 · p. 4
  • The per-frame gold standard mixes leave-one-subject-out Abramson KDE, a uniform prior, dataset centerbias, and fine-tuned SalFoM predictions, with weights selected by held-out-subject fixation likelihood.The KDE captures within-frame spatial consistency, while SalFoM accounts for semantic and cross-frame or video consistency.Source: §3.2 · pp. 4–5
  • The evaluation uses log-likelihood at gaze points, information gain over centerbias, and information gain explained relative to the gold-standard headroom.The paper also reports KLD, CC, SIM, AUC, and NSS; the gold standard enters only the IGE denominator, so LL, IG, and model rankings do not depend on it.Source: §3.2 · p. 5

4 DATASET ANALYSIS AND RESULTS

SalTempto reveals a much larger gain from temporal modelling than LEDOV, while substantial headroom remains and observed gaze patterns expose behaviors current models miss.

  • 45 percentage points separate SalFoM-Tuned from DG-MR-Tuned on SalTempto, versus 19 pp on LEDOV; removing temporal context leaves 33% of SalTempto headroom recovered, versus 58.5% with context.On LEDOV, the same ablation leaves 64% recovered, with only 15% of SalFoM-Tuned’s information gain over centerbias depending on temporal context.Source: §4 · p. 5
  • Fine-tuning raises SalFoM by about four IGE points on each SalTempto split and lifts DG-MR from at or below centerbias to 13%.The authors interpret these gains as adaptation to the recording setup, not newly acquired capability; remaining headroom is not explained by too few training fixations.Source: §4 · p. 5
  • 0.77 bit (≈41.5% of SalTempto’s headroom above centerbias) remains unexplained by SalFoM-Tuned, compared with less than 25% on LEDOV.SalTempto’s mean per-frame residual is +0.77 bit versus +0.56 bit on LEDOV; its 95th-percentile residual is +2.4 bit versus +1.67 bit, with difficulty distributed across frames rather than concentrated in outliers.Source: §4 · pp. 5–6
  • Zero-shot model rankings correlate across LEDOV and SalTempto (Spearman ρ = 0.83), and both datasets rank the same model families best and worst.SalFoM ranks first on each dataset, while the UNISAL variants occupy the bottom two ranks.Source: §4 · p. 5
  • SalTempto’s residual difficulty is broader and less skewed than LEDOV’s, rather than concentrated in occasional outlier frames.Residual skewness is +2.40 on SalTempto and +4.72 on LEDOV.Source: §4 · p. 6
  • Manual review of high-residual clips identifies three additional visual behaviors that current models often fail to capture, alongside previously reported failure modes.The supplied examples describe object permanence despite occlusion, scene inertia, and anticipatory saccades.Source: §4 · pp. 6–7
  • During occlusion, subjects keep fixating the rod–vice interaction while models shift toward the visible glove, hand, or other salient content.The rod–vice interaction accounts for a majority of gaze density over time, and remains interesting to observers when occluded.Source: §4 · p. 6, Figure 3 · p. 7
  • After an abrupt scene shift, human gaze takes a few seconds to reset, while model predictions move to the new visual content unrealistically fast.The example is a fencing bout in which the fencers shift position within a few frames.Source: §4 · p. 6

5 DISCUSSION

SalTempto examples expose temporal gaze behaviors that current models miss, including anticipatory saccades, scene inertia, and center-anchored gaze at viewing onset. The discussion also notes limits in the gold standard, early-frame protocol, and scope of the benchmark comparison.

  • In the badminton example, viewers track the shuttle with both trailing and anticipatory fixations, while models focus on the players.The gaze trail spans the shuttle’s trajectory, but models miss the moving shuttle that viewers find most salient.Source: Figure 4 · p. 8
  • After a fencing scene cut, human gaze remains consistent with salient pre-cut regions, while models quickly reattend to current foreground figures.The effect perturbs the model-based gold standard, so residual-to-gold underestimates the architectural shortfall.Source: Figure 5 · p. 8
  • The gold standard includes dataset-fine-tuned SalFoM, but substituting other panel models changes neither model ranking nor the cross-dataset contrast.SalFoM recovers the smallest fraction of headroom under its own ceiling, and a more capable future model can replace it.Source: §5 · p. 8
  • The gold standard is an improving target: as models improve, more available gaze information becomes measurable and the explanatory bar rises.Source: §5 · p. 9
  • At video onset, viewers’ gaze is center-anchored for about 15 frames, but SalFoM, ViNet-E, and DeepGaze MR fail to capture this pattern.The authors retain these frames to preserve the empirical centerbias and avoid a baseline-versus-model mismatch; they treat the failure as specific to recording setups.Source: §5 · p. 9, Figure 6 · p. 9
  • SalTempto targets foreground human activity, so conclusions do not cover dynamics without an acting agent.Its validation and test splits each contain 10 one-minute videos; the dataset-level comparison is against LEDOV alone.Source: §5 · p. 9

6 CONCLUSION

The conclusion finds that LEDOV remains weak for diagnosing temporal attention, while SalTempto reveals gains from temporal models and substantial remaining failures. The authors identify three missed gaze behaviors and leave their frequency and contribution to residual error for future annotation.

  • On LEDOV, a fine-tuned static probe recovers more than half the headroom above centerbias, while the strongest temporal model improves only modestly.Source: §6 · p. 9
  • On SalTempto, the static probe recovers only a small fraction of headroom, and the strongest temporal model leaves nearly half unexplained.The temporal model also shows a heavier-tailed per-frame residual than on LEDOV.Source: §6 · p. 9
  • LEDOV remains useful for ranking general saliency architectures but is weak as a diagnostic for temporal attention.Source: §6 · p. 9
  • The described behaviors are object permanence despite occlusion, scene inertia, and anticipatory saccades, alongside failure modes categorized by Tangemann et al. (2020).How often each occurs and how much residual each explains require a dedicated annotation study left to future work.Source: §6 · pp. 9–10
  • The results indicate that the field has measured architectural progress against a benchmark that does not measurably require temporal reasoning.SalTempto provides a setting in which the field can begin to address this mismatch directly.Source: §6 · p. 10

AI USE STATEMENT

The authors disclose generative AI use in implementation, editing, drafting, and code tasks, while specifying tasks for which it was not used. They report reviewing AI-assisted work and taking responsibility for the final content.

  • Coding agents assisted with implementation, environment setup, and replication of public baselines from GitHub.Source: AI USE STATEMENT · p. 10
  • AI tools helped edit the paper, draft from author-written notes, and create or edit software code and formatting scripts.Source: AI USE STATEMENT · p. 10
  • The authors reviewed AI-assisted text and code, took responsibility for final content, and state that AI did not design or interpret the research.They also state that evaluation-script outputs supplied all numbers.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The SalTempto eye-tracking study received institutional ethics approval, obtained participant consent, and anonymizes released gaze data. Its procedures follow standard psychophysics practice, with no known risks beyond brief screen viewing.

  • The institutional ethics board approved the eye-tracking experiment, and participants gave informed consent before recording.Source: ETHICS STATEMENT · p. 10
  • The study used chin-rest stabilization, five-point calibration, drift checks between blocks, and self-paced breaks.Source: ETHICS STATEMENT · p. 10
  • Released gaze data are anonymized, and the authors report no known risks beyond brief screen viewing.Stimuli come from the publicly available HACS-Segments dataset.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement documents the evaluation setup and points to released data and code. It also identifies the materials available for training, validation, and test.

REPRODUCIBILITY STATEMENT

  • Section 3 documents stimulus selection, recording protocol, dataset splits, model checkpoints, fine-tuning, compute, and construction of the centerbias and gold-standard mixture.App. A reports multiple metrics for each model and dataset-split pair; App. C specifies the dataset-level dynamism metrics.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The anonymised repository contains training and validation videos, raw 500 Hz gaze trajectories, per-frame binned positions, test videos with gaze withheld, and evaluation code.The hosted benchmark is stated to be released on acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10
  • All benchmarked architectures are evaluated from their publicly released checkpoints, with provenance cited in Section 3.2.Source: REPRODUCIBILITY STATEMENT · p. 10

A PER-(DATASET, SPLIT) RESULTS

The appendix defines information-theoretic evaluation against a centerbias and gold-standard reference, and reports pooled results and cross-dataset model-rank consistency.

A PER-(DATASET, SPLIT) RESULTS

  • Information Gain Explained (IGE) measures the fraction of the centerbias-to-gold-standard headroom captured by a model.It is based on log-likelihood assigned to held-out gaze, with IGE equal to 0 for centerbias and 1 for the gold standard.Source: A PER-(DATASET, SPLIT) RESULTS · p. 14
  • Table 1 compares post-fine-tuning IGE for DG-MR and SalFoM on each dataset’s validation and test splits.SalFoM is fine-tuned on each dataset’s training split; DG-MR uses its released LEDOV-trained checkpoint on LEDOV and is fine-tuned on SalTempto.Source: Table 1 · p. 14
  • Table 2 reports dataset-pooled quantities with leave-one-video-out jackknife intervals.The appendix also reports Spearman rank consistency between LEDOV and SalTempto for the 13-model panel and for a 10-model zero-shot subset.Source: A PER-(DATASET, SPLIT) RESULTS · p. 14

B CONTROL EXPERIMENTS

Control experiments test sensitivity to the gold-standard mixture and isolate SalFoM’s temporal context; a zero-shot DIEM evaluation serves as a secondary check.

B CONTROL EXPERIMENTS

  • Replacing SalFoM in the gold-standard mixture shifts the ceiling by at most 0.11 bit and the static-versus-temporal gap by at most 1.4 points on either dataset.The cross-dataset contrast is reported not to depend on which model serves as the mixture component; LL and IG are computed directly from model outputs, whereas IGE uses a panel-wide normalization.Source: B CONTROL EXPERIMENTS · pp. 14–15
  • With SalFoM’s target frame repeated across its context window, the static condition recovers 64% of headroom on LEDOV and 33% on SalTempto.Removing temporal context costs SalFoM 0.26 bit (15% of its information gain over centerbias) on LEDOV and 0.48 bit (44% of its gain) on SalTempto.Source: B CONTROL EXPERIMENTS · p. 15, Table 8 · p. 17
  • The repeated-frame static condition is a lower bound because SalFoM never saw that input during training.Even if the entire LEDOV drop were artefactual and equally large on SalTempto, at least 0.22 bit of the SalTempto drop would remain attributable to temporal information.Source: B CONTROL EXPERIMENTS · p. 16
  • The reported gold-standard sensitivity analysis compares ceiling IG and the fine-tuned models’ IGE under alternative deep mixture components.Its gap is SalFoM minus DG-MR, pooled over validation and test videos.Source: Table 7 · p. 17
  • On DIEM, zero-shot DG-MR recovers approximately 22% and SalFoM approximately 54% of gold-standard headroom.DIEM uses 17 test videos with audio playback and heterogeneous recording equipment; its weak ranking correlations with LEDOV and SalTempto make it a secondary check.Source: B CONTROL EXPERIMENTS · p. 17

C DATASET-LEVEL DYNAMISM ANALYSIS

Dataset-level analyses find that gaze information transfers poorly between SalTempto frames, unlike LEDOV, supporting the distinction in within-video gaze dynamism.

C DATASET-LEVEL DYNAMISM ANALYSIS

  • On LEDOV, cross-frame transfer LL is +1.06 versus centerbias +1.26 bits per fixation on validation; on SalTempto it is −0.06 versus +1.31.On test, the corresponding values are +1.21 versus +1.33 on LEDOV and +0.09 versus +1.31 on SalTempto.Source: C DATASET-LEVEL DYNAMISM ANALYSIS · p. 17, Table 9 · p. 18
  • On SalTempto validation, cross-frame recovery is non-positive because transfer LL falls below the uniform baseline, so another frame’s gold density can mislead rather than transfer usefully.This analysis frames temporal tracking of salient regions as necessary for accurate predictions on SalTempto.Source: C DATASET-LEVEL DYNAMISM ANALYSIS · pp. 17–18
  • Per-frame gold-density symmetrised KL is about 30% larger on SalTempto, while matched-resolution optical-flow magnitude is about 2.5× larger than on LEDOV.The flow comparison is partly confounded by camera shake in HACS-derived web video; flow variability is more concentrated in a few LEDOV videos than across SalTempto.Source: C DATASET-LEVEL DYNAMISM ANALYSIS · p. 18, Table 9 · p. 18

Related papers

Loading 2610.03276v1…