Source-linked AI summary
Eye Gaze Tells You Where to Compute: Gaze-Driven Efficient VLMs
Qinyu Chen, Jiawen Qi
TL;DR
Redundant visual tokens make VLM inference costly for resource-constrained edge devices, while existing efficiency approaches have limitations in architecture, activations, or alignment with task-relevant regions. GazeVLM uses human gaze to select ROIs and pair them with a coarse global view. It reduces visual tokens by up to 93.1% and FLOPs by up to 50%, while maintaining or improving answer quality against full-image inference.
Problem
Redundant visual tokens make VLM inference costly for resource-constrained devices, where vision-related computation can exceed 90% of image-captioning cost.
Source: §1 · pp. 1–2Method
GazeVLM is a training-free framework that uses eye gaze to crop task-relevant ROIs and combines them with a low-resolution global view for inference.
Source: §1 · p. 2, §4.2 · p. 5Results
Up to 93.1% fewer visual tokens and 50% fewer FLOPs accompany maintained or improved answer quality relative to full-image inference.
Source: §6 · p. 9Takeaways & Limitations
The results highlight the promise of aligning model attention with human perceptual signals to enhance interpretability and robustness.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) deliver impressive performance in understanding visual content with language instructions. However, redundancy in vision tokens results in the degenerated inference efficiency of VLMs, which hinders real-time use on edge consumer devices such as AR/VR devices. Existing efficiency methods commonly prune visual tokens using learned saliency, sparse attention schedules, or controller policies, but they often require architectural modification or access to intermediate activations. These pipelines add inference-time modules that increase compute and memory and often lead to an accuracy trade-off. Moreover, they also suffer from misalignment between the prompts and the region of interest in the images. Without human guidance, the model may focus on the wrong regions and miss small, high-frequency details when prompts or scenes change. In this paper, we propose GazeVLM, a training-free framework that uses the human eye gaze as a natural supervisory signal to allocate computation where it matters. By extracting gaze-driven regions of interest (ROIs) and optionally combining them with a low-resolution global view, GazeVLM mimics fovea-periphery perception to cut redundant visual tokens while preserving task-relevant details. We evaluate the visual question answering tasks on Qwen2.5-VL-3B/7B on the VOILA-COCO benchmark with human gaze. Quality of the answer is assessed by GPT-4o pairwise judging and a weighted score over coverage, accuracy, details, and fluency. Efficiency is measured by token counts and FLOPs. GazeVLM reduces visual tokens by up to 93.1%, total tokens by up to 59.6%, and FLOPs by 50%, while keeping better answer quality relative to full-resolution baselines. Our results show that aligning model computation with human gaze offers a simple, plug-and-play path toward efficient VLM inference on consumer devices.
1 INTRODUCTION
VLM inference is costly for resource-constrained AR/VR devices, while eye tracking offers a user-specific signal for directing computation. GazeVLM uses that signal to reduce visual tokens while maintaining or improving answer quality against full-image inference.
1 INTRODUCTION
- Visual tokenization can make vision computation more than 90% of image-captioning cost, while the associated energy use strains battery-powered AR/VR devices.A cited estimate puts a 1024×1024 image at 765 tokens and over 500 J for a 7B model.Source: §1 · pp. 1–2
- Gaze provides real-time, user-specific intent and a natural supervisory signal for multimodal inference.Recent AR/VR devices are equipped with or moving toward eye-tracking modules.Source: §1 · p. 2
- 93.1% fewer visual tokens on Qwen2.5-VL-3B and 75.6% fewer on Qwen2.5-VL-7B accompany pairwise win rates of 53.4% and 50.8% against full-image inference.GazeVLM is training-free and plug-and-play.Source: §1 · p. 2, Figure 1 · p. 2
2 RELATED WORKS
Prior work reduces visual-token cost through token selection or compression, while physiological-signal work has generally emphasized alignment and user experience. GazeVLM instead uses gaze to guide efficient inference without retraining.
2 RELATED WORKS
- Visual-token inference cost grows quadratically with token count, motivating more efficient processing strategies.VLMs commonly represent images as sequences of visual tokens.Source: §2 · p. 2
- Prior methods use language instructions, multimodal similarity, event priors, or projection-based compression, but may degrade performance or rely solely on vision or text cues.Examples include SparseVLM, MMTok, Event-priori VLM, and OmniVLM.Source: §2 · pp. 2–3
- Physiological-signal research has focused mainly on alignment and user experience, whereas GazeVLM uses gaze to guide efficient inference without retraining.The authors describe gaze as a direct signal for selecting ROIs while preserving semantic coverage.Source: §2 · p. 3
3 INSIGHTS
Human gaze reflects sparse, task-dependent attention: the relevant foreground shifts with the reasoning goal. The paper’s insight is to align visual processing with gaze to reduce redundant tokens while retaining informative content.
3 INSIGHTS
- People fixate on a small part of the visual field, and gaze is associated with perception, reasoning, and fixations that can precede answers.The cited cognitive-science findings connect gaze with objects people are reasoning about.Source: §3 · p. 3
- The task-relevant foreground shifts with the reasoning goal, unlike VLMs’ uniform processing of foreground and background.Figure 2 illustrates attention shifts across contexts.Source: §3 · p. 3, Figure 2 · p. 3
- Aligning model processing with gaze is proposed to reduce redundant vision tokens while preserving informative content for multimodal reasoning.The overview describes gaze-driven ROIs with an optional low-resolution global view.Source: §3 · p. 4, Figure 3 · p. 4
4 METHOD
GazeVLM uses gaze to identify attended image regions and form an adaptive input for a base VLM. It combines a high-resolution ROI with an optional coarse global view to retain detail and scene context.
4 METHOD
- GazeVLM is a training-free token-compressed inference method.The method uses gaze to guide computation allocation.Source: §4 · p. 4
4.1 GAZE-DRIVEN ROI EXTRACTION
- Gaze-driven ROI extraction smooths sparse gaze traces into a heatmap, selects pixels covering a fixed proportion of gaze mass, and encloses them in a bounding box.A minimum crop size guards against overly tight crops and sparse-gaze degeneracies; the resulting crop preserves relevant details while discarding irrelevant regions.Source: §4.1 · pp. 4–5
4.2 TWO-SCALE INPUT REPRESENTATION
- The two-scale input pairs a low-resolution global view for scene layout with a gaze-driven ROI for fine detail, then combines their visual tokens with the text query.The model is guided to prioritize the ROI while using global context; the design aims to use tokens more effectively and improve efficiency and accuracy.Source: §4.2 · p. 5
5 EXPERIMENTS
Experiments on VOILA-COCO evaluate gaze-driven ROI inputs and a two-scale variant on Qwen2.5-VL-3B/7B, measuring answer quality and compute. ROI-only inputs reduce computation while quality varies with crop size and model; adding a global thumbnail improves quality over ROI-only at modest extra cost.
5 EXPERIMENTS
- The evaluation uses VOILA-COCO on Qwen2.5-VL-3B/7B and assesses answer quality with GPT-4o pairwise judgments and a weighted score, alongside FLOPs.For each sample, gaze is smoothed into a heatmap; the ROI is the smallest region reaching threshold ρ, and the two-scale input adds a 28×28 global view.Source: §5 · pp. 5–6
5.1 EFFECT OF ROI SELECTION ON EFFICIENCY AND ACCURACY
- At ρ = 0.05, Qwen2.5-VL-3B cuts visual tokens from 64 to 4.4 and FLOPs from 267.6 GFlops to 132.8 GFlops, with a 53.4% win rate over baseline.Its win rate peaks at 68.2% over baseline at ρ = 0.5, with 23.4 visual tokens.Source: §5.1 · p. 6
- Qwen2.5-VL-7B reaches 50.8% win rate over baseline at ρ = 0.3 with 15.6 visual tokens, after underperforming at ρ = 0.05.The passage attributes this shift to the larger model's use of global relational cues; it reports a plateau around 51–53%.Source: §5.1 · p. 6
- As ρ increases, ROI wins stay comparatively steady while ties rise and losses fall for both model sizes.The passage interprets this pattern as larger crops primarily converting losses into ties once sufficient scene structure is included.Source: §5.1 · p. 6, Figure 4 · p. 7
5.2 TWO-SCALE IUPUT: DO WE NEED TO PRESERVE GLOBAL CONTEXT?
- Across both model sizes, two-scale inputs improve on ROI-only at modest FLOPs cost while remaining far below the full-image baseline.For 3B, two-scale wins 51–57% over ROI across the threshold sweep; for 7B, it wins 50–53%.Source: §5.2 · p. 7, Figure 5 · p. 8
5.3 QUALITATIVE RESULTS
- In qualitative cases, both ROI-only and two-scale inputs succeed where baseline fails in case 1; in case 5, only two-scale answers correctly.The other cases are generally ties among baseline and GazeVLM.Source: Figure 7 · p. 9
6 CONCLUSION
GazeVLM uses gaze-driven regions of interest with a low-resolution global view to reduce visual computation. On VOILA-COCO with Qwen2.5-VL-3B/7B, it cuts visual tokens and FLOPs while maintaining or improving answer quality relative to full-image inference.
6 CONCLUSION
- Visual tokens fall by up to 93.1% and FLOPs by up to 50%, while answer quality is maintained or improved relative to full-image inference.Experiments use VOILA-COCO with Qwen2.5-VL-3B/7B.Source: §6 · p. 9
- Gaze-driven regions of interest combined with a low-resolution global view emulate fovea-periphery perception, directing computation to relevant image content.The framework is training-free and plug-and-play.Source: §6 · p. 9
- The results highlight the promise of aligning model attention with human perceptual signals for interpretability and robustness.The authors identify streaming gaze, diverse modalities, and broader real-world AR/VR applications as future directions.Source: §6 · p. 9