Source-linked AI summary
NTIRE 2026 Challenge on Video Saliency Prediction: Methods and Results
Andrey Moskalenko, Alexey Bryncev, Ivan Kosmynin, Kira Shilovskaya, Mikhail Erofeev, Dmitry Vatolin, Radu Timofte, Kun Wang, Yupeng Hu, Zhiran Li, Hao Liu, Qianlong Xiang, Liqiang Nie, Konstantinos Chaldaiopoulos, Niki Efthymiou, Athanasia Zlatintsi, Panagiotis Filntisis, Katerina Pastra, Petros Maragos, Li Yang, Gen Zhan, Yiting Liao, Yabin Zhang, Yuxin Liu, Xu Wu, Yunheng Zheng, Linze Li, Kun He, Cong Wu, Xuefeng Zhu, Tianyang Xu, Xiaojun Wu, Wenzhuo Zhao, Keren Fu, Gongyang Li, Shixiang Shi, Jianlin Chen, Haibin Ling, Yaoxin Jiang, Guoyi Xu, Jiajia Liu, Yaokun Shi, Jiachen Tu
TL;DR
Video saliency prediction requires scalable methods and data for estimating human attention in video. This challenge introduces an openly licensed 2,000-video dataset with crowdsourced annotations and evaluates solutions using established metrics; the final methods rely on large pretrained video backbones and multi-scale spatiotemporal modeling.
Problem
Collecting eye-tracking data for video saliency is expensive and difficult to scale, motivating improved crowdsourced annotation and automatic prediction methods.
Method
The challenge constructs a 2,000-video dataset with crowdsourced mouse-tracking annotations and evaluates submitted saliency predictors using four established metrics and code review.
Results
The final solutions were built on large pretrained video backbones and multi-scale spatiotemporal modeling, with 7 teams completing the code-sharing phase.
Takeaways & Limitations
The challenge dataset and described methods support future research toward more accurate video saliency prediction models.
Abstract
from arXiv · showhide
This paper presents an overview of the NTIRE 2026 Challenge on Video Saliency Prediction. The goal of the challenge participants was to develop automatic saliency map prediction methods for the provided video sequences. The novel dataset of 2,000 diverse videos with an open license was prepared for this challenge. The fixations and corresponding saliency maps were collected using crowdsourced mouse tracking and contain viewing data from over 5,000 assessors. Evaluation was performed on a subset of 800 test videos using generally accepted quality metrics. The challenge attracted over 20 teams making submissions, and 7 teams passed the final phase with code review. All data used in this challenge is made publicly available - https://github.com/msu-video-group/NTIRE26_Saliency_Prediction.
1. Introduction
Video saliency prediction models how attention is distributed across visual scenes, supporting multimedia applications. The challenge uses scalable crowdsourced mouse tracking to address the cost and difficulty of collecting eye-tracking data.
- Visual saliency prediction approximates human visual attention by estimating which regions of a scene attract viewers.
- Saliency modeling supports perceptually guided compression, quality evaluation, content retargeting, media enhancement, immersive video, and 3D mesh simplification.
- Video saliency methods extend spatial modeling with temporal information to capture motion across frames.
- Modern neural approaches learn spatial, temporal, and audio patterns directly from data, while newer work adapts large-scale vision-language models.
- Crowdsourced mouse tracking makes large-scale saliency annotation more feasible than expensive eye-tracking, and this challenge further refined collection and post-processing.
2. Video Saliency Prediction Challenge
The challenge provides an openly licensed, large-scale video saliency dataset with crowdsourced annotations and evaluates submissions using established metrics and reproducible code review. Its 2,000 videos are split into training and test subsets for staged public and private evaluation.
- Challenge Dataset: The source videos were diverse YouTube content distributed under an open CC-BY license.
- Challenge Dataset: 2,000 videos and over 1M frames were collected from more than 5,000 viewers, with over 70 viewers per video on average.The average video length is 18 seconds.
- Evaluation: Evaluation used CC, SIM, AUC-Judd, and NSS, with final ranking determined by average rank across the four metrics.
- Challenge Dataset: The dataset contains 1,200 training videos with fixation data and saliency maps, alongside an 800-video test set.
- Evaluation: The private test subset contains 500 videos, and final-phase methods were assessed on the entire subset after code and prediction verification.
3. Results
The challenge results compare seven code-reviewed solutions using multi-metric ranking on the private test subset. The leading approaches combine large pretrained video backbones with complementary or multi-scale spatiotemporal modeling.
- The ranking used mean rank across metrics on the private test subset, with ties resolved by the first differing metric in priority order.
- The iLearn team ranked first with a shared InternVideo2 backbone and two complementary decoders whose predictions were fused.
- The CVSP, ARK MMLAB, Vertex, AAM, SHU-MIIPLab, and NTR teams submitted distinct approaches spanning predictive representations, hierarchical fusion, audio-visual modeling, diffusion, and dual streams.
4. Teams and Methods
The participating methods use pretrained video representations with architectures that model spatial detail, temporal dynamics, multi-scale features, or complementary expert predictions. The described systems include ensemble, predictive, hierarchical, and prior-enhanced decoding strategies.
- 4.1. iLearn: iLearn combines a shared InternVideo2 backbone with two complementary decoders and fuses their predictions into final saliency maps.
- 4.1. iLearn: The first iLearn expert uses temporal modulation, 3D residual blending, FiLM conditioning, and a learnable center-bias prior.
- 4.1. iLearn: The second iLearn expert uses multi-scale fusion, temporal gating, and deep auxiliary supervision for intermediate saliency predictions.
- 4.2. CVSP: PredJSal is motivated by predictive coding and V-JEPA2’s self-supervised masked-video prediction representations.
- 4.2. CVSP: PredJSal repurposes V-JEPA2 representations, refines temporal context with self-attention, and decodes features using 3D convolutions and multi-scale skip connections.
4.3. ARK MMLAB
ARK MMLAB uses a hierarchical architecture that combines multi-level spatiotemporal features with temporal attention and top-down decoding. Its bidirectional enhancement of TMFI adds bottom-up aggregation to exploit both semantic and fine-grained information.
- The pipeline combines a visual encoder, feature upsampling module, temporal attention mechanism, and hierarchical decoder.
- InternVideo2-1B extracts four feature levels, from fine-grained local details to abstract high-level semantics.
- Temporal attention derives weights from the deepest feature and applies them across all levels to emphasize salient frames.
- The decoder progressively upsamples and concatenates features from deep to shallow branches before producing 224 × 224 saliency maps.
4.4. Vertex
Vertex extends TMFI’s top-down feature enhancement with a bottom-up pathway and combines predictions from multiple complementary models. The resulting design enriches multi-scale representations and applies smoothing to improve spatial and temporal coherence.
- TMFI extracts four-scale spatiotemporal features, enhances them top-down, and decodes fused saliency maps.
- Vertex adds bottom-up max-pooling aggregation to TMFI, aligning and combining features with element-wise addition and 3D convolutions.
- The bidirectional design expands the representation from 4 to 8 feature scales and generates eight saliency maps for learnable weighted fusion.
- Vertex fuses predictions from DiffSal, SalFoM, and TMFI using a weighted combination.
- Gaussian and temporal smoothing are applied after fusion to improve spatial coherence and reduce temporal inconsistencies.
4.5. AAM
AAM models human attention as a shared latent process across modalities, scenes, and time. It combines multimodal encoders, relevance-gated fusion, temporal dynamics, hierarchical hyperbolic representations, and geometry-aware decoding.
- AAM represents attention through hierarchical semantic specialization and temporal dynamics across modalities, scenes, and time.
- Frozen DINOv3, CLIP, and Wav2CLIP encoders provide visual, textual, and audio representations, with audio mapped into visual semantic space.
- Relevance-gated cross-attention lets audio cues contribute when they are semantically aligned with visual content.
- The Fokker–Planck Dynamics module models the evolution of frame-wise attention over a spatiotemporal manifold.
- Hierarchical entailment learning imposes semantic ordering in hyperbolic space, and a geometry-aware decoder returns spatial attention maps to Euclidean space.
4.7. NTR
NTR uses pretrained video and image streams to jointly model motion and appearance, then decodes their multi-scale features into saliency maps. The architecture includes temporal pooling, progressive fusion, a Gaussian center prior, and multi-objective training.
- The dual-stream network combines spatiotemporal motion cues from a video stream with fine-grained spatial appearance cues from a frame stream.
- The video stream uses an R(2+1)D-18 backbone on L=8 frames and temporally aggregates four-scale features with learned softmax weights.
- The frame stream uses ImageNet-pretrained ConvNeXt-Tiny features to provide complementary static-scene context.
- The decoder fuses both streams across four scales through projections, concatenation, convolutions, and bilinear upsampling.
- A Gaussian center prior is fused into the decoder output before sigmoid prediction and upsampling to the original resolution.
- Training combines KL divergence, correlation, NSS, and fixation BCE losses, while two-stage optimization uses a lower backbone learning rate.
4.8. Baseline Solution
The organizers’ baseline creates a Gaussian Center Prior from the average saliency maps of the training videos and applies it uniformly across test-video frames.
- 1,200 training-video saliency maps were averaged to create the Center Prior target.A Gaussian centered in the frame was then fitted to this average.
- The Gaussian fitting optimized only the σx and σy parameters using L2 loss.
- The resulting Center Prior was duplicated across all frames and supplied as a sample submission.
5. Conclusion
The challenge introduced a large crowdsourced video-saliency dataset and found that top solutions used pretrained video backbones with multi-scale spatiotemporal modeling.
- 2,000 videos were released with saliency annotations collected from more than 5,000 assessors through crowdsourced mouse tracking.
- 7 teams proposed solutions and passed the final code-sharing phase.
- Top solutions used large pretrained video backbones and multi-scale spatiotemporal modeling.
- The challenge dataset and described methods are intended to support future research on more accurate video saliency prediction models.
6. Teams and Affiliations
The supplied passage identifies a member of the challenge paper, but provides no team or affiliation information.
- The passage lists Konstantinos Chaldaiopoulos as a paper member.