PaperlayerView exact revision on arXiv

Source-linked AI summary

GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths

Xianyu Chen, Ming Jiang, Qi Zhao

arXiv:2408.02788v1cs.CV

TL;DR

Scanpath models predict when and where people look, but do not explain what or why lies behind each fixation. GazeXplain annotates fixation explanations and jointly predicts scanpaths and language; experiments report superior performance in both tasks.

  • Problem

    Traditional scanpath models predict when and where people shift attention but do not explain what or why underlies each fixation.

    Source: §1 · pp. 1–2

  • Method

    GazeXplain annotates fixation-level explanations across three eye-tracking datasets and jointly predicts scanpaths and language using an attention-language decoder, semantic alignment, and cross-dataset co-training.

    Source: §1 · pp. 2–3

  • Results

    GazeXplain demonstrates superior performance in scanpath prediction and explanation; on AiR-D, semantic alignment raises BLEU-4 from 27.7 to 30.7 and CIDEr-R from 97.3 to 123.1.

    Source: §4.4 · p. 12, §5 · p. 14

  • Takeaways & Limitations

    Fixation explanations offer a way to inspect what observers attend to and why, with potential for enhancing human-machine interaction.

    Source: §1 · p. 1

    LLaVA-generated explanations can be rephrased for fixations in the same region; manual corrections addressed less than 0.58% of outliers.

    Source: Analyses on the Generated Explanations… · p. 28

Abstract

from arXiv · show

While exploring visual scenes, humans' scanpaths are driven by their underlying attention processes. Understanding visual scanpaths is essential for various applications. Traditional scanpath models predict the where and when of gaze shifts without providing explanations, creating a gap in understanding the rationale behind fixations. To bridge this gap, we introduce GazeXplain, a novel study of visual scanpath prediction and explanation. This involves annotating natural-language explanations for fixations across eye-tracking datasets and proposing a general model with an attention-language decoder that jointly predicts scanpaths and generates explanations. It integrates a unique semantic alignment mechanism to enhance the consistency between fixations and explanations, alongside a cross-dataset co-training approach for generalization. These novelties present a comprehensive and adaptable solution for explainable human visual scanpath prediction. Extensive experiments on diverse eye-tracking datasets demonstrate the effectiveness of GazeXplain in both scanpath prediction and explanation, offering valuable insights into human visual attention and cognitive processes.

1 Introduction

Scanpath models predict when and where people shift attention, but do not explain what or why lies behind each fixation. GazeXplain introduces fixation-level language explanations, a joint prediction model, and cross-dataset training.

1 Introduction

  • Existing scanpath models predict when and where attention shifts but do not explain what or why motivates individual fixations.The paper identifies this lack of explainability as a barrier to understanding and applying the models in real-world scenarios.Source: §1 · p. 2
  • GazeXplain introduces a task that jointly predicts scanpaths and explains them in natural language, with ground-truth fixation-level annotations on three public eye-tracking datasets.The annotations provide a basis for modeling explanations of visual attention.Source: §1 · p. 2
  • Its general architecture uses an attention-language decoder to jointly predict scanpaths and explanations, with semantic alignment and cross-dataset co-training.The alignment mechanism promotes consistency between fixations and explanations; co-training targets generalization beyond task-specific datasets.Source: §1 · pp. 2–3
  • Experiments across diverse datasets demonstrate effectiveness in generating accurate scanpaths and explanations, and examine the contributions of explanation prediction, semantic alignment, and co-training.The introduction presents these components as relevant to model performance.Source: §1 · p. 3

2 Related Work

Related work covers dynamic scanpath prediction, natural-language explanation in other domains, and vision-language models; GazeXplain brings these lines together for fixation explanations.

2 Related Work

  • Prior scanpath methods use heuristics, statistical priors, supervised learning, or reinforcement learning, but lack interpretability and struggle to explain predicted fixations [56] [100].Dynamic scanpath prediction is described as relatively underexplored compared with static saliency prediction [56].Source: §2 · p. 3
  • Deep-learning natural-language explanations span classification, visual reasoning, recommendation, and sentiment analysis; this work applies them to eye-tracking for the first time.GazeXplain jointly predicts scanpaths and explanations to establish a semantic connection between them.Source: §2 · p. 3
  • Vision-language models generate natural-language descriptions of visual information [60] [65], but scanpath models had not fully leveraged them for attention prediction; GazeXplain uses them for explanation.The paper also draws on these models for ground-truth annotations and language modeling.Source: §2 · pp. 3–4

3 GazeXplain

GazeXplain pairs scanpath prediction with fixation-level natural-language explanations to address the limited transparency of existing scanpath models. It introduces annotations across diverse eye-tracking tasks and a model combining a vision-language encoder, attention-language decoder, semantic alignment, and cross-dataset co-training.

3 GazeXplain

  • GazeXplain addresses the limited transparency of scanpath models by jointly predicting scanpaths and generating natural-language explanations for fixations.The paper also introduces fixation-level explanation annotations across eye-tracking datasets.Source: §3 · p. 4

3.1 Data

  • The annotations provide fixation-level explanations across free-viewing, object search, and VQA, unlike image-wise descriptions such as captions or visual stories.They cover AiR-D, OSIE, and COCO-Search18, including target-present and target-absent subsets.Source: §3.1 · pp. 4–5
  • LLaVA generates one-sentence descriptions from images marked with a red circle around each fixation, followed by manual quality control.Reviewers remove circle references, standardize length, verify recognized text, and correct inconsistent descriptions within fixation clusters.Source: §3.1 · pp. 4–5, Figure 2 · p. 5
  • 7,004 images and 86,407 fixations across three eye-tracking datasets receive explanations averaging 10.66 ± 3.54 words per fixation.Table 1 presents statistics for the annotated datasets.Source: §3.1 · p. 6, Table 1 · p. 6

3.2 Model

  • A vision-language encoder represents image content and task instructions, while cross-dataset co-training uses varied task contexts to support generalization.The model is trained across datasets with different task designs.Source: §3.2 · pp. 6–7, Semantic Alignment · pp. 8–9

Attention-Language Decoder.

  • The attention-language decoder predicts fixation distributions and generates an explanation for each fixation in the predicted scanpath.Its attention decoder uses temporal-step features and cross-attention with joint vision-language features to predict spatiotemporal fixation distributions.Source: Attention-Language Decoder · p. 7

Semantic Alignment.

  • Semantic alignment compares explanation, fixation, and multimodal similarities with visual-similarity pseudo-labels to encourage consistency across fixations.The training objective combines this loss with scanpath prediction and language generation losses.Source: Semantic Alignment · p. 8

4 Experiments

Experiments evaluate GazeXplain on VQA, free-viewing, and visual-search eye-tracking data using scanpath, saliency, and language metrics. Results report gains in scanpath prediction and explanation quality, with ablations and qualitative examples examining the model’s components and outputs.

4.1 Experimental Setup

  • The evaluation combines AiR-D VQA, OSIE free-viewing, and COCO-Search18 target-present and target-absent data with scanpath, saliency, and language metrics.The compared models include SaltiNet, PathGAN, IOR-ROI, ChenLSTM, and Gazeformer.Source: §4.1 · p. 9

4.3 Ablation Study for Scanpath Prediction and Explanation

  • On AiR-D, adding fixation-based explanations without semantic alignment raises SM from 0.356 to 0.378, while CIDEr-R improves from 66.7 to 97.3 over the BLIP captioner baseline.The ablation examines the language decoder (EXP), semantic alignment (ALN), and cross-dataset co-training (CT).Source: Language Decoder · p. 10, §4.3 · p. 10, Table 3 · p. 11

Language Decoder.

  • On AiR-D, semantic alignment raises CIDEr-R from 97.3 to 123.1 and improves SM from 0.378 to 0.386 and CC from 0.647 to 0.662.The study describes alignment as linking fixation semantics with their corresponding explanations.Source: Language Decoder · p. 10, §4.3 · p. 10

4.2 Scanpath Prediction Results

  • GazeXplain consistently surpasses state-of-the-art methods across datasets on scanpath prediction, with results spanning spatial, temporal, and semantic gaze measures.The reported saliency metrics also indicate strong performance at highlighting visually important image regions.Source: §4.2 · p. 10, Table 2 · p. 11

Cross-Dataset Co-Training.

  • Cross-dataset co-training raises SM from 0.346 to 0.386 and CIDEr-R from 115.1 to 123.1 by training one model across datasets.The reported results describe gains across datasets and identify the approach as a way to use shared information from diverse data sources.Source: Cross-Dataset Co-Training · pp. 11–12

4.4 Scanpath Explanation Results

  • Explanation evaluations report language-metric gains from semantic alignment across datasets, with co-training especially effective on OSIE free-viewing and COCO-Search18 target-absent scenarios.For AiR-D, BLEU-4 rises from 27.7 to 30.7 and CIDEr-R from 97.3 to 123.1; the faithfulness analysis reports AUC increases from 0.878 to 0.902 and from 0.871 to 0.902 for alignment and co-training, respectively.Source: §4.4 · pp. 12–13
  • Semantic alignment increases COCO-Search18 target-absent vocabulary size from 566 to 685 words and UnP% from 66.04% to 71.35%, while explanation lengths remain consistent.Co-training also consistently boosts UnP% and helps balance vocabulary sizes and explanation lengths across datasets.Source: Table 4 · p. 12, §4.4 · p. 13

4.5 Qualitative Analysis

  • In qualitative examples, GazeXplain predicts task-relevant fixations and gives more accurate, specific descriptions than Gazeformer with BLIP captions.Examples include identifying a cake, a dog, and a trash can, while Gazeformer captions misdescribe scene details; the figure shows scanpaths and explanations for two key fixations in each row.Source: §4.5 · pp. 13–14, Figure 5 · p. 14

5 Conclusion

GazeXplain introduces a scanpath explanation task, fixation-level annotations, and a model that predicts scanpaths while generating natural-language explanations. Experiments report strong performance in both tasks and motivate the use of explanations in language-driven user environments.

5 Conclusion

  • GazeXplain defines a scanpath explanation task and provides ground-truth explanations across eye-tracking datasets.Its attention-language decoder jointly predicts scanpaths and generates natural-language explanations, using semantic alignment to keep fixations and explanations consistent.Source: §5 · p. 14
  • Cross-dataset co-training leverages diverse training datasets to enhance the model’s generalizability.The conclusion also describes this approach as addressing data- and task-specific biases.Source: §5 · p. 14
  • Experiments demonstrate strong performance in scanpath prediction and explanation, and the conclusion identifies language-driven user environments as a possible application for GazeXplain’s explanations.Source: §5 · p. 14

1 Introduction

The supplementary materials provide model and training details, further ablations, and additional quantitative and qualitative comparisons. These results are reported as supporting GazeXplain’s performance and adaptability across visual tasks.

1 Introduction

  • The supplementary materials detail the GazeXplain model, including its vision-language encoding module and objective functions, and provide implementation and training settings.Source: §1 · p. 22
  • Supplementary ablations evaluate the language decoder, semantic alignment, and cross-dataset co-training on AiR-D, OSIE, and COCO-Search18.Source: §1 · p. 22
  • Additional quantitative and qualitative comparisons examine vision-language models, prompt settings, competitor training strategies, and scanpath methods; the reported results emphasize performance on OSIE free-viewing and COCO-Search18 visual search.Source: §1 · pp. 22–23

2 Supplementary Method

GazeXplain encodes visual content and task instructions, then uses a combined objective to train scanpath prediction and fixation-level explanation generation.

Vision Encoding.

  • A Transformer encoder refines local CNN features by modeling relationships among them, producing task-independent visual features.The CNN features capture localized image details, while the Transformer adds a holistic scene representation.Source: Vision Encoding · p. 23

Language Encoding.

  • The language encoder processes task instructions into features that capture their semantic meaning and intent.Instructions can be general questions or questions specifying a search target.Source: Language Encoding · pp. 23–24

Multimodal Integration.

  • GazeXplain concatenates visual and instruction features into a joint representation for subsequent scanpath prediction and explanation generation.The visual features capture bottom-up input, while the instructions provide task-specific, top-down guidance.Source: §2.1 · p. 23, Multimodal Integration · p. 24

2.2 Objectives

  • For each fixation, the training data pairs position and duration with its corresponding language explanation.The scanpath loss encourages predicted fixations to resemble the ground-truth fixation sequence.Source: §2.2 · pp. 24–25
  • The final objective combines fixation, explanation, and semantic alignment losses to train scanpath prediction and fixation-level explanation generation.The alignment term encourages explanations to connect semantically with visual features associated with each fixation.Source: §2.2 · p. 24

Language Prediction Loss.

  • The language prediction loss trains explanations to remain grammatically correct and semantically consistent with the predicted scanpath and task instruction.It promotes descriptions that reflect what the model sees at each fixation.Source: Language Prediction Loss · p. 25

3 Implementation Details

Training uses supervised learning followed by self-critical sequence training, with shared settings for compared models.

3 Implementation Details

  • Training runs for eight supervised epochs, followed by two epochs of self-critical sequence training (SCST).SCST uses a linearly decaying learning rate starting at 10^-5 and batch size 8.Source: §3 · p. 25
  • Supervised training uses Adam with learning rate 4 × 10^-4, weight decay 5 × 10^-5, and batch size 16.The original dataset splits are retained, and compared models use the same settings for fairness.Source: §3 · p. 25
  • Generated scanpaths are constrained to between one and sixteen fixations.Source: §3 · p. 25

4 Supplementary Ablation Study

Ablations across AiR-D, OSIE, and COCO-Search18 report complementary benefits from explanation decoding, semantic alignment, and cross-dataset co-training.

4 Supplementary Ablation Study

  • Adding the language decoder improves scanpath prediction, spatial saliency, and explanation quality across all datasets.With co-training, SM rises by 0.01+ on OSIE and 0.02+ on all datasets, while CIDEr-R gains are 11.0 on OSIE and 30.0+ elsewhere.Source: §4 · p. 26
  • Semantic alignment improves most metrics across datasets, including CIDEr-R by about 10.0+ on each dataset and OSIE CC by 0.018.The reported improvements span scanpath and saliency metrics as well as explanation quality.Source: §4 · pp. 26–27
  • On COCO-Search18 Target-Present, co-training raises SM from 0.449 to 0.480 and CIDEr-R from 133.2 to 139.6.The study reports overall score increases across all metrics on most datasets.Source: §4 · p. 27
  • With all three components, GazeXplain achieves superior scanpath, saliency, and explanation performance across diverse datasets.The study characterizes language decoding as empowering explanation, semantic alignment as fostering coherence, and co-training as promoting generalizability.Source: §4 · p. 27

5 Supplementary Quantitative Results

Supplementary analyses compare generated descriptions with human annotations, test observer-answer settings, assess competitor cross-dataset training, and evaluate GazeXplain on additional datasets.

Analyses on the Generated Explanations from Large Vision-Language

  • LLaVA descriptions score CIDEr-R 110.4 and SenS 0.606, versus GPT-4V at 99.1 and 0.592; GazeXplain scores 106.3 and 0.590.The authors describe LLaVA's descriptions as reasonably accurate and GazeXplain's description ability as similar; LLaVA outputs can vary for fixations in the same region, with manual corrections addressing outliers under 0.58%.Source: Analyses on the Generated Explanations… · pp. 27–28

Exploration of Observer Answer.

  • On AiR-D, removing observer answers only at test time yields SM 0.385 and NSS 1.845, compared with 0.386 and 1.851 when answers are present.Removing answers from both training and test yields SM 0.380 and NSS 1.810, while still outperforming compared models; GazeXplain can predict observer-specific or general scanpaths.Source: Exploration of Observer Answer · p. 28, Table 2 · p. 28

Cross-Dataset Training for Competitors.

  • Directly combining training datasets lowers ChenLSTM and Gazeformer performance relative to single-dataset training, whereas GazeXplain addresses this challenge with its design and co-training strategy.Table 3 covers AiR-D, OSIE, and COCO-Search18.Source: Cross-Dataset Training for Competitors · p. 28, Table 3 · p. 29

Generalizability across Datasets.

6 Supplementary Qualitative Results

GazeXplain’s qualitative examples show it predicting human-like fixations on salient and likely target objects while generating explanations more semantically aligned with those fixations than Gazeformer. Comparisons span free-viewing and object-search scenes.

6 Supplementary Qualitative Results

  • Supplementary comparisons with Gazeformer and human ground truth span OSIE and COCO-Search18 Target-Present and Target-Absent tasks, with GazeXplain consistently improving key-object fixation prediction.The examples position GazeXplain as a potential interpretable tool for studying visual perception and attention.Source: §6 · pp. 29–30, Figure 1 · p. 30, Figure 2 · p. 31

Results on OSIE Dataset.

Results on COCO-Search18 Datasets.

  • On COCO-Search18, GazeXplain focuses on plausible target locations and links explanations to fixated objects, unlike Gazeformer’s irrelevant or nonexistent-object descriptions.Examples include plant searches near desks or nightstands and fork searches at tables.Source: Figure 2 · p. 31, Results on COCO-Search18 Datasets · pp. 31–32
Loading 2408.02788v1…