NTIRE 2026 Challenge on Video Saliency Prediction: Methods and Results

1. Introduction

• Saliency prediction estimates human attention across scenes and supports applications including compression, quality evaluation, content retargeting, enhancement, immersive content, and 3D mesh simplification.Saliency maps represent predicted attention distributions, while fixation maps are aggregated across viewers and smoothed into normalized probability maps.Source: §1 · p. 1, §1.2 · pp. 1–2

1.1. Saliency Prediction Methods

• Saliency prediction methods evolved from low-level color, contrast, and texture cues to video models incorporating motion and neural networks learning spatial, temporal, and audio patterns.Newer work is beginning to adapt large-scale Vision Language Models for saliency prediction; modeling has also extended to VR/360° content and 3D meshes.Source: §1.1 · p. 1

1.2. Video Saliency Datasets

• The largest video datasets containing eye-fixation data are Hollywood-2 [63], with 1,707 videos viewed by 19 participants, and DHF1K [102], with 1,000 videos annotated by 17 viewers.Source: §1.2 · p. 2
• The challenge uses mouse movements to collect video fixations, improving crowdsourced data collection, filtering, and post-processing to obtain high-quality saliency data.Mouse-based proxy annotation addresses the expense and scalability limits of eye tracking.Source: §1.2 · p. 2

2.1. Challenge Dataset

• The dataset contains 2,000 videos and over 1M frames, with data from over 5,000 viewers and more than 70 viewers per video.The collection procedure refined prior crowdsourced methodology [66] through participant filtering and fixation-data processing to bring data quality closer to eye-tracker data; source videos came from two public datasets [105].Source: §2.1 · p. 2

2.2. Evaluation

• Submissions were ranked by average rank across CC, SIM, AUC-Judd, and NSS on the test set, with ties resolved by the first differing metric.The final phase required predictions, a factsheet, and reproducible code, which organizers verified; 7 teams completed it, and evaluation included the Center Prior baseline.Source: §2.2 · p. 2

2.3. Challenge Phases

• The dataset was split into 1,200 training videos and 800 test videos, with public and private subsets of 300 and 500 videos, respectively.Intermediate phases evaluated each participant’s best public-subset solution on 250 private videos; final evaluation used the entire private subset. Table 1 reports mean-rank results on that private subset.Source: §2.3 · pp. 2–3, Table 1 · p. 3

3. Results

• Two teams tied in average rank; metric-priority rules select the team with the highest CC as winner, and iLearn is identified as first place.The passage says all metric values and the final ranking appear in Table 1 but provides no metric values.Source: §3 · p. 3
• iLearn’s first-place solution uses a shared InternVideo2 [103] backbone, complementary decoders, and averaged expert saliency predictions.Source: §3 · p. 3
• CVSP’s second-place solution uses a large-scale V-JEPA2 [5] backbone to estimate saliency from predictive spatiotemporal features.Source: §3 · p. 3
• ARK MMLAB’s third-place solution builds a hierarchical model on InternVideo2 [103], aligning multi-level spatiotemporal features with feature upsampling.Source: §3 · p. 3
• Vertex extends TMFI with a bottom-up aggregation path alongside its original top-down fusion for richer multi-scale feature interaction.Source: §3 · p. 3
• AAM combines audio-visual fusion, temporal-dynamics modeling, and a hyperbolic decoder to predict saliency maps.Source: §3 · p. 3
• SHU-MIIPLab predicts saliency from RGB frames and optical flow with a diffusion-based model using state-space blocks.Source: §3 · p. 3
• NTR uses a dual-stream design, combining a pretrained video backbone for motion cues with a pretrained image backbone for fine spatial details.Source: §3 · p. 3

4.1. iLearn

• The framework combines a shared InternVideo2 [103] backbone with two structurally complementary decoders and fuses their predictions into final saliency maps.The pipeline is illustrated in Figure 1.Source: §4.1 · p. 3, Figure 1 · p. 3
• InternVideo2-Stage2 6B extracts multi-level features from layers 11, 23, 35, and 47, and is fine-tuned with LoRA [35] while preserving pretrained video representations.The features span low-level spatial details to high-level semantic abstractions.Source: §4.1 · p. 3
• Expert 1 uses layer-47 features to generate temporal attention that multiplicatively modulates shallower features, followed by 3D residual blending, FiLM [72], and a learnable center-bias refinement head.The center bias leverages the tendency of human gaze toward the image center.Source: §4.1 · pp. 3–4
• Expert 2 projects all four feature levels into a unified dimension, fuses them top-down with concatenation and 3D residual blocks, and applies deepest-level temporal gating.Independent saliency heads at intermediate levels provide deep auxiliary supervision and dense gradient signals.Source: §4.1 · p. 4
• The experts target complementary scene types, and their predictions are combined by averaging inverse-sigmoid logits before projecting back to probabilities with sigmoid.Expert 1 uses spatial priors for central subjects and stable dynamics; Expert 2 is more adaptable to off-center or multiple salient regions.Source: §4.1 · p. 4
• Each expert is trained independently in two stages: first train the decoder with the backbone frozen, then jointly fine-tune decoder and LoRA modules at a lower learning rate.The objective combines KL divergence, Pearson’s correlation coefficient, similarity, and normalized scanpath saliency losses; ensemble fusion occurs at inference.Source: §4.1 · p. 4

4.2. CVSP

• V-JEPA2 [5] motivates the backbone choice because its predictive-coding training supports representations of intuitive physics, including object permanence and shape consistency [23].The framework links saliency to prediction errors in a model of how the physical world behaves.Source: §4.2 · p. 4
• PredJSal repurposes a V-JEPA2 Giant (ViT-g/16) backbone pretrained with self-supervised masked video prediction to produce video saliency maps.The encoder extracts a dense spatiotemporal feature volume and intermediate representations at three evenly spaced layers.Source: §4.2 · p. 4
• A temporal self-attention module refines encoder features at each spatial position, while a 3D convolutional decoder upsamples them and reduces time to one output frame.Multi-scale skip connections add intermediate encoder features at three decoder stages to recover spatial detail, and a lightweight 2D convolutional head produces the final prediction.Source: §4.2 · p. 4, Figure 2 · p. 4
• The model is trained with four random seeds, and its predicted saliency maps are averaged at inference to produce one map.This is intended to reduce the effect of random initialization.Source: §4.2 · p. 4

4.3. ARK MMLAB

• The framework combines a visual encoder, feature upsampling module, temporal attention mechanism, and hierarchical decoder to predict video saliency.Its hierarchical architecture is designed to capture spatial semantics and temporal dynamics from video clips.Source: §4.3 · p. 5

4.3.1. Visual Encoder

• InternVideo2-1B [104] encodes each 3×16×224×224 clip into four feature levels, reshaped as 1408-channel 5D tensors at spatial resolutions 112×112, 56×56, 28×28, and 14×14.The levels represent fine-grained details at early stages and higher-level semantics at deeper stages.Source: §4.3.1 · p. 5

4.3.2. Feature Upsampling Module

• The feature upsampling module aligns feature levels using bilinear interpolation, 2D convolutions, Group Normalization, and GELU activations.It resizes y1, y2, and y3 to 128×128, 64×64, and 32×32 with 512, 512, and 768 channels, respectively, while retaining y4 at 14×14.Source: §4.3.2 · p. 5

4.3.3. Temporal Attention and Hierarchical Decoder

• Temporal attention derives frame weights from the deepest feature using 3D convolutions and temporal pooling, then applies them across all four feature levels before decoding.The decoder processes four branches top-down, using 3D convolutions and ReLU activations, upsampling and concatenating each with the adjacent shallower branch before producing a 224×224 saliency map.Source: §4.3.3 · p. 5

4.4. Vertex

• TMFI uses Video Swin Transformer [56] features, top-down semantic enhancement, and a hierarchical decoder whose saliency maps are fused by 3D convolution.It extracts four-scale spatiotemporal features before producing the final prediction.Source: §4.4 · p. 5
• Vertex adds bottom-up max-pooling aggregation to TMFI’s top-down pathway, aligning and adding features before 3D-convolution refinement and expanding the representation from four to eight scales.The design combines high-level semantics with low-level details.Source: §4.4 · p. 5
• Vertex generates eight multi-scale saliency maps and replaces TMFI’s 3D-convolution fusion with learnable weighted aggregation.The passage describes this fusion as more efficient and stable.Source: §4.4 · p. 5
• Vertex combines predictions from DiffSal [109], SalFoM [65], and TMFI [120], then applies Gaussian and temporal smoothing to improve spatial coherence and reduce frame-to-frame inconsistencies.The models’ outputs are fused using a weighted combination.Source: §4.4 · pp. 5–6

4.5. AAM

• AAM encodes video with adapted DINOv3 [86], text with CLIP [77], and audio with Wav2CLIP [107], gating audio-visual cross-attention so audio contributes only when semantically aligned.The audio representation is mapped into the visual semantic space.Source: §4.5 · p. 6
• AAM’s Fokker–Planck Dynamics module models the evolution of frame-wise attention over a spatiotemporal manifold through drift, diffusion, and correction.Figure 5 depicts the module’s attention transition along the video frame axis.Source: §4.5 · p. 6, Figure 5 · p. 6
• AAM lifts visual and textual representations into hyperbolic space to enforce semantic hierarchy, then decodes the structured representations into spatial attention maps.Figure 5 describes entailment cones for general-specific attention ordering and a geometry-aware decoder projecting back to Euclidean space.Source: §4.5 · p. 6, Figure 5 · p. 6

4.6. SHU-MIIPLab

• SSF-DiffNet is a denoising diffusion model that takes an RGB frame and its Flow-Anything optical-flow map as inputs.Source: §4.6 · p. 6
• The network combines a saliency-map denoising network with a state-space-model-based condition-generation network.Source: §4.6 · p. 6
• SAM2/Hiera encoders with adapters extract RGB and flow features; SI-MB injects RGB semantics into motion cues, and SA-MB aligns appearance and motion features to form denoising conditions.Source: §4.6 · p. 6
• The denoising network reuses SI-MB to transfer texture cues and semantic priors from the conditions to the noisy mask during iterative saliency-map optimization.Source: §4.6 · p. 6

4.7.1. Dual-Stream Network Architecture

• The proposed pretrained dual-stream saliency network jointly exploits spatiotemporal motion cues from video and fine-grained spatial appearance cues from frames, leveraging large-scale video and image priors without additional saliency-data pretraining.Source: §4.7.1 · p. 6

4.7.2. Video Stream

• The video stream processes 8-frame clips with R(2+1)D-18 pretrained on Kinetics-400 [41] [94], then aggregates features across time at four spatial scales.The (2+1)D factorization separates spatial 2D and temporal 1D convolutions; learned temporal softmax weights form each scale’s spatial feature map.Source: §4.7.2 · pp. 6–7

4.7.3. Frame Stream

• The frame stream processes the center frame with ConvNeXt-Tiny pretrained on ImageNet-1K to provide four stages of complementary static-scene context.These appearance features complement the motion-focused R(2+1)D features.Source: §4.7.3 · p. 7

4.7.4. Decoder with Center Prior Fusion

• The decoder fuses both streams across four scales using learned 1×1 projections, concatenation, 3×3 convolutions, and a Gaussian center prior.It proceeds from coarse to fine scales and bilinearly upsamples the output to the original video resolution.Source: §4.7.4 · p. 7

4.7.5. Loss Function

• Training combines KL divergence, correlation, normalized scanpath saliency, and fixation BCE losses, weighted 1.0, 0.7, 0.3, and 0.1, respectively.The fixation BCE uses positive class weight 8.0.Source: §4.7.5 · p. 7

4.7.6. Warm-Start Training Schedule

• Two warm-start training stages use AdamW with cosine annealing and a backbone learning rate 0.1× the head rate to preserve pretrained features.Each stage warm-starts from the previous best checkpoint.Source: §4.7.6 · p. 7

4.7.7. Inference

• Inference predicts each frame from a centered 8-frame clip at stride 2 and 224×224 resolution, averaging predictions from original and horizontally flipped clips.Maps are upsampled, min-max normalized per video, and written as grayscale MP4 at the original FPS.Source: §4.7.7 · p. 7

4.8. Baseline Solution

• The Center Prior was fitted to the average saliency maps from 1,200 training videos using a frame-centered Gaussian, optimizing σx and σy with L2 loss.Source: §4.8 · p. 7
• The resulting Center Prior was duplicated across all frames of the test videos and provided to participants as a sample submission.Source: §4.8 · p. 7

• Top solutions used large pretrained video backbones and multi-scale spatiotemporal modeling.Source: §5 · p. 7
• The challenge dataset contains 2,000 videos with saliency annotations collected from more than 5,000 assessors through crowdsourced mouse tracking.Source: §5 · p. 7
• Seven teams proposed solutions and passed the code-sharing phase in the final phase.The authors expect the methods and dataset to support research toward more accurate video saliency prediction models.Source: §5 · p. 7