PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowing When to Trust a Prior: Reliability-Gated Cue Fusion for Video Gaze Prediction

Lichen Zhu, Yueqian Lin, Yiheng Wang, Hai "Helen" Li, Yiran Chen

arXiv:2610.08663v1cs.CV

TL;DR

Gaze-free priors may add information to video gaze prediction, but defocus reliability varies by frame and unconditional fusion can harm performance. FocusGate gates priors using frame-level defocus-map shape, improving results across supervised model–domain comparisons and converting harmful or null fusion into positive results in tested transfer domains.

  • Problem

    Defocus can mark a subject in shallow depth of field but become texture noise in deep focus, and whether gaze-free priors add value beside gaze-trained models and on which frames was untested.

    Source: §1 · p. 1

  • Method

    FocusGate rank-fuses gaze-free priors and uses a per-frame gate over three defocus-map shape statistics to select frames where the estimator is above chance on average.

    Source: §1 · p. 1, §2.2 · p. 2

  • Results

    FocusGate improves shuffled AUC in all sixteen supervised model–domain cells, with fifteen gains significant and one domain pre-registered and scored once.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The ensemble lifts supervised predictors despite trailing ViNet-S and ViSAGE alone on every domain, supporting complementarity within the tested comparisons.

    Source: §4.1 · p. 3

    The gate is specific to defocus, the only tested cue with a collapse regime; future work will seek reliability statistics for other cues.

    Source: §5 · p. 5

Abstract

from arXiv · show

Video gaze prediction is led by gaze-trained models, yet gaze-free priors carry signal those models have not absorbed, if one knows when to trust them. We propose FocusGate, a gated ensemble of gaze-free priors whose members may abstain. A per-frame gate reads three shape statistics of a defocus map and selects the frames on which the estimator is above chance on average, so rejected frames reduce to the base exactly, while midrank normalisation lets an all-zero prior abstain at zero parameters. Gated fusion is significantly positive on film, sports and web video, whereas unconditional fusion is harmful on sports and null on web. Added to four supervised predictors, the NTIRE 2026 champion among them, FocusGate improves all sixteen model-domain cells in shuffled AUC, fifteen significantly, one domain pre-registered and scored once, while adding only 1% to the champion's latency. Alone, it surpasses TASED-Net and UNISAL in shuffled AUC on film with a 16-frame causal mean.

1. INTRODUCTION

Gaze-free cues such as defocus can be informative, but their reliability changes across frames, and unconditional fusion can reduce performance. FocusGate gates cue use per frame, improving results when the estimator has a regime in which it collapses to chance.

1. INTRODUCTION

  • Defocus is a frame-varying region cue: it can mark the subject in shallow depth of field but become texture noise in deep focus.The paper asks whether such priors add value beside gaze-trained models and on which frames, noting prior blur injections were unconditional.Source: §1 · p. 1

Contribution

  • The per-frame gate reads three defocus-map shape statistics and selects frames where the estimator is above chance on average.Unconditional fusion of the scratch-trained estimator lowers sAUC on every tested base.Source: §1 · p. 1
  • Gating helps when an estimator has an identifiable regime in which it collapses to chance, not merely when it is weak.A stronger estimator shrinks that regime, while domain shift can reopen it.Source: §1 · p. 1

Contributions

  • FocusGate improves shuffled AUC in all sixteen model–domain cells, fifteen significantly, while adding 1% to the NTIRE 2026 champion’s latency.Alone, it surpasses TASED-Net and UNISAL in shuffled AUC on film with a 16-frame causal mean.Source: §1 · p. 1
  • On film, gating converts scratch-estimator harm from −0.94† to +0.04, and on frozen sports transfer flips −1.40† to +0.68∗.The gate equals its base on rejected frames, and midrank normalisation keeps an all-zero prior inert at zero parameters.Source: §1 · p. 1

2. METHOD

FocusGate combines a rank-summed base of gaze-free priors with a defocus cue that enters only when a per-frame gate accepts it. The gate uses map shape, while rank fusion allows cues to be combined on a common scale.

2. METHOD

  • Eight gaze-free priors are rank-summed into base B, while defocus map D enters only through gate c_t, which never sees fixations.The overview depicts the gate opening on bimodal D and closing on unimodal D, where prediction equals B.Source: §2 · pp. 1–2, Figure 1 · p. 2
  • The recipe defocus estimator adds an ImageNet-pretrained MobileNetV3-L encoder, train-time depth distillation, and a high-frequency input channel to a scratch U-Net.Both predict per-pixel in-focus probability; the scratch model has 2.76M parameters and the recipe model reaches AUROC 0.958/0.953 on clean/degraded held-out images.Source: §2 · p. 2

2.1. Shallow-DoF reliability gate

  • The shallow-DoF gate accepts a frame when spread ≥ 0.55, separation ≥ 0.45, and the sharp-area fraction is 3%–70%.Its thresholds were fixed on CUHK before gaze data was used and never retuned.Source: §2.1 · p. 2

2.2. Gated rank-fusion

  • Per-frame rank normalisation maps cues to [0, 1] with tied values sharing their midrank, making heterogeneous cues scale-free.The weighted sum is not ranked again; when c_t=0, the prediction is exactly base B.Source: §2.2 · p. 2

3. EXPERIMENTAL SETUP

Evaluation uses held-out videos and one shared protocol, with shuffled AUC as the reported metric. The setup also specifies frozen transfer domains, confirmatory DIEM scoring, and video-clustered uncertainty estimates.

Data

  • Hollywood-2 uses the standard 823/884 split, while transfer to UCF-Sports and DHF1K keeps every parameter frozen before scoring.DIEM is a confirmatory fourth domain, acquired after parameters were frozen and pre-registered for a single scoring.Source: §3 · pp. 2–3

Leakage-free subsets

  • The four supervised baselines are scored on held-out video only; ViSAGE runs zero-shot, and the held-out sets span Hollywood-2, UCF-Sports, DHF1K, and DIEM.TASED-Net and ViNet-S use 32-frame clips, while UNISAL uses a recurrent state warmed over up to 96 frames.Source: §3 · p. 3

Metrics

  • Shuffled AUC is the sole reported metric, with negatives drawn from other videos’ fixation distributions so centre bias alone scores chance.Scores are per-video means averaged over videos and marginalised over twenty negative-sampling seeds.Source: §3 · p. 3
  • Uncertainty uses 10,000-resample percentile bootstrap intervals clustered on videos; significance marks 95% confidence intervals excluding zero.The primary contrast was pre-specified, while later arms were each specified, train-tuned, and scored once with unadjusted intervals.Source: §3 · p. 3

4. RESULTS

FocusGate improves shuffled AUC for supervised predictors across the tested model–domain cells, while results also show when its defocus gate helps, how it transfers, and where its use is bounded. The gate is useful where a cue has a regime that collapses to chance, and frozen sports and web results remain positive under transfer.

4.1. Comparison with prior work

  • FocusGate improves sAUC in all sixteen supervised model–domain cells, with fifteen gains surviving Holm correction and ViSAGE on web within noise.The four models include ViSAGE, the NTIRE 2026 champion.Source: §4.1 · p. 3
  • With a 16-frame causal mean, FocusGate surpasses TASED-Net and UNISAL in sAUC on film and untouched DIEM.Its single-frame sports score is 79.3 against ViSAGE’s 80.5; adding it to the champion costs 1% in latency.Source: §4.1 · pp. 3–4
  • Shared-midrank ranking lets a constant face-prior map self-abstain, worth +0.6 points at frozen weights versus +0.12 for the explicit face gate.Tie-splitting rank normalisation would instead fabricate a full-contrast pattern.Source: §4.1 · p. 4

4.2. When does the gate help?

  • On the reference base, gating converts scratch-estimator fusion from −0.94† ungated to +0.04 [−0.12, +0.21] gated.The gated result is a null; the paired gated-minus-ungated contrast is +0.99 [+0.74, +1.24].Source: §4.2 · p. 4
  • Defocus maps fall to chance when unimodal and rise above 60 when bimodal, whereas sharpness and motion never approach chance.A better defocus estimator shrinks but does not remove the collapse regime; gating still adds +0.60∗, though the film gain is within noise.Source: §4.2 · p. 4, Figure 2 · p. 4

4.3. Frozen transfer and controls

  • Under frozen transfer, sports fusion changes from −1.40† ungated to +0.68∗ gated, while web changes from −0.08 to +0.57∗.Controls find the gain is not an artifact of frame selection and is present in the map’s spatial content.Source: §4.3 · p. 4

5. CONCLUSION

FocusGate is a gated ensemble of gaze-free priors, and its defocus gate selects frames where the estimator is above chance on average. It improves supervised predictors across tested domains, while the gate remains specific to defocus.

5. CONCLUSION

  • FocusGate is a gated ensemble of gaze-free priors in which each prior may abstain.Its defocus gate uses three map-shape statistics to select frames where the estimator is above chance on average.Source: §5 · p. 5
  • Added to four supervised predictors, FocusGate improves every model’s sAUC in every tested domain, including a pre-registered one.The gate is specific to defocus, the only tested cue with a collapse regime.Source: §5 · p. 5

6. COMPLIANCE WITH ETHICAL STANDARDS

The study uses public eye-tracking data and collected no new human or animal data; ethical approval was therefore not required.

6. COMPLIANCE WITH ETHICAL STANDARDS

  • This retrospective study used public eye-tracking data and collected no new human or animal data, so no ethical approval was required.The datasets were Hollywood-2 with Actions-in-the-Eye recordings, UCF-Sports, DHF1K, and DIEM.Source: §6 · p. 5

Supplementary Material

Supplementary results report intervals and controls for the main comparisons, including all-frame DHF1K scoring and the sixteen supervised-model comparisons. Under the all-frame protocol, FocusGate ranks fixations better than TASED-Net but scores lower on AUC-Judd.

Supplementary Material

  • The supplement reports intervals, controls, and negative results omitted from the four-page limit, with values generated from the result files under the stated protocol.That protocol averages per-video means and uses twenty negative-sampling seeds and 10,000-resample video-clustered percentile bootstraps.Source: Supplementary Material · p. 7

A. All-frame protocol on DHF1K 601–700

  • On all annotated frames of 100 DHF1K validation videos, the gate accepted 54.9% of frames; FocusGate ranks fixations better than TASED-Net but scores lower on AUC-Judd.The protocol uses shuffled-AUC negatives from the other 99 videos’ fixations.Source: A. All-frame protocol on DHF1K 601–700 · p. 7

B. The sixteen “+ FocusGate” cells of Table 1 with intervals

  • Fifteen of sixteen supervised model–domain gains survive Holm correction; the ViNet-S–DIEM cell was added after DIEM pre-registration.The table reports paired per-video deltas, 95% cluster-bootstrap intervals, and bootstrap p-values.Source: B. The sixteen “+ FocusGate” cells of… · p. 7

C. Table 2 with intervals

The table gives two delta estimates with 95% intervals for cue–estimator rows, including a frozen sports transfer.

C. Table 2 with intervals

  • Defocus, recipe (40%): +0.36 [+0.05, +0.67]∗ and +0.60 [+0.44, +0.76]∗.The excerpt does not identify which delta column corresponds to each value.Source: C. Table 2 with intervals · p. 7
  • Defocus, classical (0%): −1.88 [−2.16, −1.61]† and +0.00 [+0.00, +0.00]; scratch (38%): −0.94 [−1.25, −0.64]† and +0.04 [−0.12, +0.21].The excerpt does not identify which delta column corresponds to each value.Source: C. Table 2 with intervals · p. 7
  • The frozen sports row (38%) lists −1.40 [−2.70, −0.13]† and +0.68 [+0.07, +1.36]∗.The excerpt does not identify which delta column corresponds to each value.Source: C. Table 2 with intervals · p. 7
  • Laplacian sharpness (17%) lists −0.67 [−0.86, −0.48]† and −0.18 [−0.26, −0.11]†; motion and depth rows are also reported.Motion (56%): +0.68 [+0.45, +0.91]∗ and +0.03 [−0.13, +0.20]; depth (MiDaS, 100%): +1.95 [+1.69, +2.23]∗ for both estimates.Source: C. Table 2 with intervals · p. 7

D. Gate controls (H2 test split, recipe estimator unless stated)

The controls compare the real gate with random and soft alternatives and test whether gains persist across motion levels and map recalibrations.

D. Gate controls (H2 test split, recipe estimator unless stated)

  • The paired gated-minus-ungated contrast is +0.23 [−0.02, +0.49] for recipe and +0.99 [+0.74, +1.24]∗ for scratch.The recipe interval includes zero, while the scratch contrast is marked significant.Source: D. Gate controls (H2 test split,… · p. 8
  • At the same 39.7% accept rate, random gates yield +0.04 to +0.25 versus base, while the real gate yields +0.60.The real gate exceeds random gates by a mean/minimum of +0.45/+0.35; accepted frames score +1.54 [+1.12, +1.96]∗ versus base.Source: D. Gate controls (H2 test split,… · p. 8
  • The soft logistic gate differs from the hard gate by −0.01 [−0.05, +0.04], and gains are significant in both extreme motion quartiles: +0.33 and +0.89.The accept rate rises from 30% to 52% across motion quartiles, but the gain persists at both extremes.Source: D. Gate controls (H2 test split,… · p. 8
  • Across 15 affine map recalibrations, gated deltas range from +0.52 to +0.64 with all lower bounds above zero.Gate decisions agree for 92.8% of adjacent frames and 82.3% of frames 16 apart.Source: D. Gate controls (H2 test split,… · p. 8

E. Density-sensitive metrics on the film test split

The section distinguishes ranking from density metrics, reports variants that did not help, and describes the fitted-scalar setup.

E. Density-sensitive metrics on the film test split

  • sAUC is the paper’s metric because centre-bias scores chance; calibrated rank-sum maps lift NSS, CC, and SIM without changing AUC-Judd, while supervised models remain ahead on density.AUC-Judd is identified as a ranking metric; the density metrics reflect a train-fitted monotone calibration.Source: E. Density-sensitive metrics on the… · p. 8

F. Variants that did not help (fitted on train, frozen sets scored once)

  • DINOv2-registers large as a DINO cue scores 77.73, with deltas of −0.50 [−0.73, −0.28]†, −0.61 [−1.14, −0.03]†, and −0.52 [−1.03, −0.01]†.Flip test-time averaging scores 78.43, with deltas +0.01 [−0.02, +0.04], −0.06 [−0.13, −0.00], and −0.02 [−0.07, +0.03]; neither variant was adopted.Source: F. Variants that did not help (fitted… · p. 8
  • Other rejected variants include per-cue temporal pooling, entropy-weighted fusion, and a DINO-internal failure detector whose best train ROC-AUC is 0.56.The uncertainty-term weight fitted to zero; the detector was below the 0.6 threshold for fitting a rule, and V-JEPA 2 maps were not fitted as cues.Source: F. Variants that did not help (fitted… · p. 8

G. Fitted scalars

  • Coordinate ascent fits weights and exponents on train sAUC; supervised fusion uses one train-frozen weight per model: 0.4, 0.3, 0.15, and 0.2.The supervised weights correspond respectively to TASED-Net, UNISAL, ViNet-S, and ViSAGE.Source: G. Fitted scalars · p. 9

H. Implementation details

The implementation specifies estimator training, gate thresholds, cue processing, fusion fitting, and scoring; the supplement illustrates how midrank enables a face prior to abstain.

H. Implementation details

  • Defocus estimators use CUHK blur-detection images, with a shared seeded split, no gaze-dataset images, and image degradation applied during training.The scratch U-Net and recipe MobileNetV3-Large estimators use different architectures and losses; both train for 40 epochs, with one run each and no fixed training seed.Source: H. Implementation details · p. 9
  • The gate thresholds three statistics of a 96×56 in-focus map: spread ≥0.55, separation ≥0.45, and sharp fraction between 0.03 and 0.70.Thresholds were set by hand on CUHK validation maps before gaze data and left unchanged; gains were flat over the reported spread/separation ranges.Source: H. Implementation details · p. 9
  • The cue recipe fixes frame and map resolutions and specifies distinct processing for face, spectral-residual, motion, flow, depth, DINO, and defocus maps.The face map is midrank-normalised and all-zero when no face is detected; DINO attention is blurred and reranked.Source: H. Implementation details · p. 10
  • Fusion scalars are selected by coordinate ascent on Hollywood-2 train sAUC, then supervised fusion weights are chosen per model and frozen.The training objective uses 823 videos and twenty negative seeds; temporal variants replace selected maps with causal-mean ranks.Source: H. Implementation details · p. 10
  • Scoring averages per-video sAUC over videos and negative-sampling seeds, with paired video-level bootstrap intervals and Holm correction over 16 cells.Timing uses one A100, batch size one, and one forward pass per predicted frame.Source: H. Implementation details · p. 10

I. Abstention under the two rank normalisations

  • When face detection fails, midrank maps the all-zero prior to a constant 0.5, leaving its fused term inert; splitting ties instead fabricates contrast.The figure contrasts a detected-face frame with a no-detection frame; split ties produce standard deviation 0.29 across the map.Source: Figure S1 · p. 11

Related papers

Loading 2610.08663v1…