Source-linked AI summary
Evaluating Visual Realism of AI-Generated Videos with Eye Tracking
Danqing Shi, Lan Jiang, Jake Lyons, Miri Zilka
TL;DR
Explicit human judgments of AI-generated videos reveal viewers’ conclusions but not the perception behind them. This paper studies two-pass eye tracking and introduces GazeEval to predict perceived visual realism. GazeEval reaches 71.3% accuracy, and the study releases its data.
Problem
Explicit judgments capture viewers’ conclusions about AI-generated videos but offer limited insight into the perception that leads to them.
Source: §1 · p. 2Method
The study conducts two-pass Visual Turing Tests and trains GazeEval to predict perceived video realism from eye-tracking data.
Source: §1 · p. 2Results
71.3% accuracy was achieved by GazeEval, compared with 61.4% for the stimulus-only automated quality evaluation.
Source: §1 · p. 2Takeaways & Limitations
Gaze during two-pass video watches carries informative signals about how people evaluate visual realism.
Source: §1 · p. 2The stimulus set contains only 5-second physical-scene videos without humans or narratives, and the two-pass protocol differs from typical viewing.
Source: §5.1 · p. 12
Abstract
from arXiv · showhide
Human evaluation of AI-generated videos has so far relied on explicit human feedback, such as judgments or ratings. While these signals capture the conclusions viewers reach, they fail to offer insights into the perception that leads to them. This paper investigates eye tracking as a complementary human evaluation signal for AI-generated videos. We conducted two Visual Turing Test (VTT) experiments with a total of 90 participants on videos generated by latest generative models, in which participants watched each video twice and then gave their feedback. Across the experiments, we collected 4,286 valid eye-tracking trials on 438 videos. Our analysis indicates that gaze patterns offer informative evaluation signals that align with explicit human feedback. Building on the findings, we propose the first computational method GazeEval to predict the perceived visual realism of AI-generated videos from eye-tracking data. It achieves 71.3% accuracy in prediction, providing evidence that eye tracking data carries informative signals about how people evaluate a video. We release the study data to support further research on eye tracking for evaluating AI-generated video.
1 Introduction
The paper asks whether eye tracking can complement explicit judgments of AI-generated video realism, addressing the limited insight those judgments provide into viewers’ perception. Two-pass Visual Turing Test studies examine gaze patterns and introduce GazeEval, a predictor of perceived realism from eye-tracking data.
1 Introduction
- Existing automated metrics correlate weakly with human perception, while explicit judgments reveal viewers’ conclusions but not the perception behind them.The paper therefore investigates gaze as an implicit evaluation signal alongside explicit human feedback.Source: §1 · p. 2
- The two-pass study protocol was designed to test whether gaze shifts from content saliency on the first watch toward anomaly search on the second.The first-pass gaze patterns were more consistent with bottom-up saliency, while the second pass involved more informative eye movements.Source: §1 · p. 2
- GazeEval is presented as the first computational method to predict perceived visual realism of AI-generated videos from human eye-tracking data.The authors collected a broader video set and explicit feedback to train models intended to generalize across video generative models.Source: §1 · p. 2
- 71.3% accuracy was achieved by GazeEval, compared with 61.4% for the stimulus-only automated quality evaluation.The study also reports that saliency evaluation metrics were more informative than eye-movement statistics, and self-consistency between watches was key to prediction.Source: §1 · p. 2
- The two Visual Turing Test experiments involved 90 participants, and the released dataset contains 4,286 valid eye-tracking trials with explicit feedback.The contribution also identifies gaze consistency across the two watches as a dominant prediction feature.Source: §1 · p. 2
2 Related Work
The related work reviews automated video-generation evaluation, human evaluation protocols, and eye tracking in generative AI. It situates this study amid weak alignment between automated metrics and human judgments, variation in human-evaluation practices, and limited evidence about gaze during viewing of generated videos.
2 Related Work
- The review spans computational evaluation, human protocols, and research linking generative AI with eye tracking.These strands frame the paper’s investigation of gaze as a complementary evaluation signal.Source: §2 · p. 2, §2.3 · p. 3
2.1 Evaluating AI-generated Videos
- Automated video metrics and benchmark suites are scalable, but their scores correlate weakly with human perceptual judgments.The paper positions human evaluation as the focus of its work.Source: §2.1 · pp. 2–3
2.2 Human Evaluation for Video Generation
- Human-evaluation studies vary in metrics, rating protocols, annotator recruitment, and prompt sets, raising reproducibility and reliability concerns.Existing protocols still rely on explicit judgments and provide limited insight into moment-by-moment perception.Source: §2.2 · p. 3
2.3 Generative AI Meets Eye Tracking
- Prior gaze studies report differences for text, images, and face-swapped videos, but do not establish how gaze relates to realism in modern generated videos.The paper identifies the gaze–cognitive-process relationship during real versus AI-generated video viewing as insufficiently understood.Source: §2.3 · p. 3
3 Visual Turing Test with Eye Tracking
The two-pass Visual Turing Test pairs explicit video judgments with eye tracking to examine how realism shapes viewing and whether gaze patterns provide evaluation signals. AI-generated clips elicited more consistent gaze across passes, while real clips prompted broader scanning; gaze also shifted from bottom-up saliency on the first pass toward less predictable viewing on the second.
3.1 Overview
- The study pairs explicit judgments with implicit eye tracking to capture moment-by-moment visual processing during the same videos.The video Visual Turing Test adapts the real-versus-AI classification task and has participants watch each clip twice before judging.Source: §3.1 · p. 3
3.2 Experiment Design
- Physics-IQ supplied 60 moving real videos, each paired with a Veo 3.1 version; in each session, participants watched 20 clips—10 real and 10 AI-generated—twice and judged each.No participant saw both versions of the same scene.Source: §3.2 · pp. 3–4
3.3 Data Collection
- Filtering retained 763 of 800 possible participant-clip trials, yielding 1,526 viewing trials across 78 clips from 40 scenes.A retained trial required both passes, at least four recorded seconds per pass, and at least five fixations inside the stimulus.Source: §3.3 · p. 4
3.4 Eye Tracking Data Analysis
- Analysis summarized eye movements with four viewing statistics and compared duration-weighted human attention maps with each other and SalFoM predictions.Attention maps used in-stimulus fixations across the five-second trial; the three comparisons were first versus second watch, and each watch versus SalFoM.Source: §3.4 · pp. 4–5
3.5 Findings
- AI clips elicited more cross-pass gaze consistency, while real clips prompted wider scanning and AI clips shorter saccades and scanpaths; differences appeared on the first watch.All saliency metrics showed significant cross-pass consistency differences in the same direction; the authors describe this as the experiment’s most consistent effect.Source: §3.5 · p. 5, Figure 2 · p. 6
- Mean CC fell from 0.600 to 0.547 on AI clips and from 0.591 to 0.525 on real clips; mean EMD rose from about 76 px to 89 and 91 px.Agreement with SalFoM was generally lower on the second pass, including SIM, CC, NSS, and AUC for both stimulus types.Source: §3.5 · p. 5, Figure 2 · p. 6
- 78.4% human-judgment accuracy accompanied group patterns that replicated 15 of 18 significant judgment-based results in stimulus-based comparisons.The remaining three results showed similar but non-significant trends.Source: §3.5 · p. 5, Figure 2 · p. 6
4 GazeEval: From Eye Tracking to Perceived Visual Realism
GazeEval tests whether eye-tracking patterns can predict the visual realism viewers perceive in AI-generated videos. The study examines gaze-rating associations, builds a gaze-based prediction pipeline, and evaluates how prediction changes with feature sets and viewer counts.
4.1 Data Collection
- 50 new participants watched videos from five generators, producing 2,760 valid eye-tracking trials, including 2,343 viewer–clip pairs from 300 AI-generated videos.The realism label averaged five-point ratings, assigning the maximum score when a viewer mistook an AI clip for real. The five generators did not include Veo 3.1, which had been used in the earlier experiment.Source: §4.1 · pp. 6–7
4.2 Do Gaze Agreement Metrics Track Perceived Realism?
- Gaze consistency correlated modestly with lower perceived realism, with correlations between 0.1 and 0.22 in magnitude and the strongest associations for viewers’ self-consistency.Agreement metrics correlated negatively with ratings and distance metrics positively; second-pass comparisons with SalFoM were consistently stronger than first-pass comparisons.Source: §4.2 · p. 7, Figure 5 · p. 9
4.3 Predicting Perceived Visual Realism from Eye Tracking Data
- GazeEval averages per-viewer scores from three attention-map comparisons into 24 video-level features for predicting whether a clip looked real.The comparisons are first versus second watch and each watch versus SalFoM; the model uses scene-grouped folds to keep all five renderings of a source scene on the same side of each split.Source: §4.3 · pp. 7, 9
4.4 Results
- 68.7% accuracy and 70.2% balanced accuracy came from combining gaze and VBench features; its 0.805 ROC-AUC exceeded gaze-only features by 0.030.The combined model’s accuracy was within one standard deviation of the best model, and stimulus features contributed a smaller but consistent amount of information alongside gaze.Source: §4.4 · p. 10
- 0.496 ROC-AUC and 0.503 accuracy left eye-movement statistics alone below the trivial baseline; adding them to the best saliency model reduced accuracy from 0.713 to 0.650.The authors attribute this contrast with within-scene findings to raw fixation and saccade measures being dominated by content differences across scenes, while saliency metrics are normalized against each clip.Source: §4.4 · pp. 10–11
- 71.3% accuracy, 0.713 balanced accuracy, and 0.775 ROC-AUC were achieved by GazeEval’s best gaze-only model, exceeding the stimulus-only baseline by 11 accuracy points and 0.11 ROC-AUC.Across five held-out folds, the best model used all 24 saliency evaluation metrics. Its prediction probabilities rank videos by perceived realism, but overlap substantially between rating classes.Source: §4.4 · p. 9, Table 1 · p. 11
- 0.751 ROC-AUC made first-versus-second-watch self-consistency the strongest single saliency-feature block.Combining all three saliency blocks added 2.6% accuracy and 0.024 AUC over the best individual block; this self-consistency feature requires two viewings.Source: §4.4 · p. 11
4.5 How Many People Are Needed to Predict Visual Realism from Eye Tracking Data?
- 71.7% accuracy was reached with eight viewers per video, compared with around 56% accuracy and 0.62 ROC-AUC for one viewer.Accuracy increased and standard deviation decreased as viewer counts rose; the curve had not converged, and the authors caution that results with only a few viewers may be unreliable.Source: §4.5 · p. 11
5 Conclusion and Discussion
The study finds that gaze patterns reflect perceived realism and that two-pass viewing supports GazeEval’s prediction of realism. Its conclusions are bounded by the video set, viewing protocol, saliency model, and omission of temporal eye-movement information.
5 Conclusion and Discussion
- GazeEval reached 71.3% accuracy, compared with 61.4% for stimulus-only automated quality evaluation, establishing gaze as a complementary channel to explicit feedback.Across 90 participants, the study collected 4,286 valid trials on 438 videos from six generative models.Source: §5 · p. 11
- Accuracy rose from 56.0% with one viewer to 71.7% with eight, although some videos had fewer than eight viewers.At k=7 and k=8, clips averaged 6.88 and 7.60 viewers, respectively.Source: Figure 7 · p. 12
- The released eye-tracking dataset is intended to support further investigation and human-centered evaluation of generative video models.Source: §5 · p. 12
5.1 Limitations and Future Work
- Findings are limited to five-second physical-scene videos without people or narratives and a two-pass protocol focused on AI detection.The authors suggest larger-scale crowdsourcing in real-world settings to address these boundaries.Source: §5.1 · p. 12
- SalFoM was trained on natural videos, so degraded predictions on AI-generated clips may confound comparisons with human attention.The authors call for validation against human attention maps from free viewing of AI-generated videos.Source: §5.1 · p. 12
- The analysis models spatial eye-tracking information but omits temporal patterns, such as the sequence and duration of visits to regions.The authors propose temporal or event-based metrics as future additions that may improve realism prediction.Source: §5.1 · p. 12
A Full Group Comparison Statistics
Figure 8 presents saliency-map comparisons and eye-movement statistics across AI-versus-real and judged-AI-versus-judged-real groups. The text says the findings are discussed in Section 3.
A Full Group Comparison Statistics
- The figure’s findings are discussed in Section 3.Source: A Full Group Comparison Statistics · p. 14
- Figure 8 compares AI versus real clips and judged-AI versus judged-real clips across saliency-map metrics and eye-movement statistics.Panels a–c show eight saliency metrics for three attention-map comparisons; panels d–e show four eye-movement statistics for each pass.Source: Figure 8 · p. 14
- Brackets in Figure 8 mark Wilcoxon signed-rank tests, with * for p<.05 and ** for p<.01.Source: Figure 8 · p. 14