Source-linked AI summary
EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input
Matteo Stoiber, Niels Buus Lassen
TL;DR
Token budgets constrain egocentric video understanding with MLLMs, while gaze-based cropping traditionally requires dedicated eye tracking. EgoGazeLite predicts gaze on-device and preserves downstream description quality relative to measured gaze across all ten evaluated cases.
Problem
MLLM egocentric video systems need to reduce visual-token budgets without losing task-relevant detail, but gaze-guided cropping traditionally depends on dedicated eye-tracking hardware.
Method
EgoGazeLite uses a lightweight dual-process predictor that fuses bottom-up visual saliency with top-down fixation-history attention for on-device gaze-guided cropping.
Results
Equivalence was confirmed in all ten downstream comparisons between predicted- and ground-truth-gaze crops across two MLLMs, three automated metrics, and two LLM judges.
Takeaways & Limitations
Predicted gaze can substitute for hardware-measured gaze in gaze-conditioned egocentric video understanding while the full gaze-and-crop pipeline runs in real time on consumer accelerator hardware.
Takeaways & Limitations
The predictor and downstream substitution were evaluated in-distribution across seven Ego-Exo4D domains, leaving performance under domain shift to unseen activities untested.
Abstract
from arXiv · showhide
The use of multimodal LLMs (MLLMs) for egocentric video understanding with wearable devices is constrained by the token budget. Memory and compute cost scale with the number of visual tokens, and high-resolution video quickly becomes expensive to transmit and process at scale. Prior work (GazeLLM) addresses this by cropping the video around the camera wearer's gaze. This reduces the number of visual tokens by about tenfold while maintaining or improving the quality of full-resolution descriptions. However, this compression strategy depends on dedicated eye-tracking hardware, which is unavailable on consumer smart glasses. Building a software-only substitute poses a joint constraint: the predictor must be accurate enough to preserve downstream description quality, yet light enough to run on-device, within the power and compute budget of a smartphone. We address this with EgoGazeLite, a lightweight dual-process gaze predictor for egocentric video. Across two MLLMs, three automated metrics, and two LLM judges, predicted-gaze crops show no significant difference from ground-truth-gaze crops. Equivalence is confirmed in all ten cases. EgoGazeLite achieves this at 15.7M parameters, 6.71 GFLOPs, and runs the full gaze-and-crop pipeline end-to-end in real time (21.6 ms/frame) on consumer accelerator hardware. Together, these results remove the need for eye-tracking hardware for token-efficient, gaze-conditioned egocentric video understanding with MLLMs.
1 Introduction
EgoGazeLite addresses the visual-token bottleneck in egocentric MLLM video understanding by replacing GazeLLM’s unavailable eye-tracking hardware with lightweight on-device gaze prediction. Predicted-gaze crops match ground-truth-gaze crops downstream while enabling real-time consumer-device deployment.
- Motivation and approach: The central efficiency challenge is reducing visual-token budgets without losing task-relevant detail, because Vision Transformer memory cost scales with visual-token count, resolution, and duration.This constraint motivates gaze-conditioned compression for first-person video systems supporting description, question answering, guidance, and assistive technology.
- Motivation and approach: 90%: EgoGazeLite reduces MLLM input through gaze-guided video cropping while removing GazeLLM’s eye-tracking dependency.The approach enables gaze-guided cropping on edge devices without dedicated eye-tracking hardware.
- Motivation and approach: Roughly tenfold: GazeLLM’s gaze-centered cropping reduces visual tokens while maintaining or improving video-description quality.This establishes gaze-guided cropping as a viable token-compression strategy, but GazeLLM depends on dedicated eye-tracking hardware.
- Results: All ten cells: predicted-gaze and ground-truth-gaze crops show no significant downstream difference across two MLLMs, three automated metrics, and two LLM judges.Equivalence testing confirms differences below half the pooled standard deviation rather than merely failing to detect a difference.
- On-device deployment: 15.7M parameters and 6.71 GFLOPs: EgoGazeLite runs the full gaze-and-crop pipeline end-to-end in real time on consumer accelerator hardware.The model represents a more than 8-fold FLOP reduction compared to prior egocentric gaze models.
2 Related Work
Prior egocentric gaze prediction includes saliency-based models that struggle with task-driven attention, while gaze-guided MLLMs use measured gaze for cropping or prompting. Token merging offers a separate post-tokenization reduction strategy, leaving predicted gaze as an open substitute for eye tracking in on-device cropping pipelines.
- Egocentric gaze prediction: Egocentric gaze prediction combines bottom-up visual saliency with first-person cues, but such models struggle when gaze follows an unfolding task rather than visual prominence.First-person cues include camera motion and hand position.
- Gaze-guided multimodal LLMs: Gaze-guided systems use measured gaze to support static-scene identification, voice-query integration, visual question answering, or region-of-interest cropping.GazeGPT, G-VOILA, VQA-MHUG, and MULAN represent these uses.
- Gaze-guided multimodal LLMs: 93%: GazeVLM reduces visual tokens by up to 93% while improving answer quality through gaze-guided cropping on static images.This reduction occurs by cropping a region of interest for a vision-language model.
- Gaze-guided multimodal LLMs: EgoGazeVQA uses gaze as a prompt-side signal for longer-form question answering rather than cropping, so it does not reduce the visual token budget.Across these systems, measured gaze is treated as hardware-provided.
- Reducing tokens after tokenization: Token merging reduces token count after patch embedding by progressively combining similar tokens inside vision-transformer blocks, unlike cropping, which reduces pixels before tokenization.Its matching rule is lightweight and data-driven rather than based on a fixed spatial region.
3 Method
EgoGazeLite uses a lightweight dual-process architecture that fuses bottom-up saliency and top-down fixation-history attention to predict gaze for on-device cropping. Its method combines EfficientNet-Lite4 features, temporal differences, gated fixation transitions, staged training, and argmax-based evaluation on standardized egocentric crops.
- Architecture: EgoGazeLite fuses a bottom-up saliency heatmap from visual input with a top-down attention heatmap from recent fixation history.The dual-process design was selected over Transformer approaches because of their impracticality for on-device deployment.
- Backbone: A shared EfficientNet-Lite4 backbone processes current and previous frames, producing [B, 448, 10, 10] deep feature maps and skip-connection features.This replaces separate spatial and temporal convolutional streams and is described as the largest single source of parameter reduction relative to Huang’s design.
- Top-down path: The top-down path tracks attended feature channels, propagates them through a two-layer LSTM, and uses an online I-DT gate to distinguish ongoing fixations from saccades.The transitioned channel weights are applied to current-frame features to produce the attention heatmap.
- Fusion: The residual fusion module treats the saliency map as the base prediction and adds a learned correction in logit space before extracting gaze by argmax.Logit-space fusion can sharpen or suppress saliency according to attention-map agreement and avoids probability-space saturation.
- Bottom-up path: The bottom-up path computes ∆F = BatchNorm(Ft− Ft−1) at deep feature resolution, replacing dense optical flow with feature subtraction after the second backbone pass.The process requires no additional computational cost beyond that second backbone pass.
- Training and evaluation: Training proceeds in three stages, while evaluation reports average angular error using the argmax point convention rather than center of mass.Stage 1 trains the backbone and bottom-up path with distance-weighted binary cross-entropy; Stage 2 trains the top-down path with the bottom-up path frozen.
4 Experiments
Experiments show that EgoGazeLite is a lightweight, real-time on-device gaze-and-crop system whose predicted-gaze crops preserve downstream MLLM description quality relative to ground-truth-gaze crops. Across two MLLMs, three automated metrics, and two LLM judges, predicted and ground-truth gaze were equivalent in all ten tested cells.
- 4.1 EgoGazeLite evaluation: 15.7M parameters and 6.71 GFLOPs make EgoGazeLite substantially smaller and cheaper than two prior egocentric gaze predictors.The model uses 3.2× fewer parameters than Huang et al. and 4.5× fewer than Lai et al.; its FLOPs are reduced 8.6-fold and 14-fold, respectively.
- 4.1 EgoGazeLite evaluation: About 10× fewer visual tokens reduce phone-to-MLLM upload bandwidth, while the pipeline supports downstream sampling rates up to 30 FPS.At the one-FPS evaluation rate, cropping runs more than 40× faster than required by downstream sampling.
- 4.2 Downstream description quality: Predicted- and ground-truth-gaze crops outperform center crops on every metric for both Gemini 2.5 Flash and Gemini 2.5 Pro, while closely tracking each other.For Pro, predicted-versus-ground-truth mean differences are at most 0.002 for BLEU and ROUGE-L and 0.006 for SBERT; for Flash, they are under 0.014 and 0.016, respectively.
- 4.2 Downstream description quality: The predicted-versus-ground-truth contrast is non-significant in all ten cells, with the smallest corrected p-value 0.054 and absolute dz below 0.17.The evaluation uses paired t-tests with Holm correction across three contrasts per cell.
- 4.2 Downstream description quality: Equivalence holds for predicted versus ground-truth gaze under the primary 0.5× pooled-SD margin and the stricter 0.2× sensitivity margin.Gaze-versus-center contrasts show substantial effects, with pHolm <.001 in 16 of 20 cells and dz ranging from 0.19 to 0.68.
- 4.2 Downstream description quality: +0.81 is the overall clip-level Spearman correlation between GPT-4o and Claude rankings, showing judge choice changes scores but not crop-condition order.The correlation ranges from +0.72 to +0.89 and is significant at p < 10−23.
5 Conclusion
EgoGazeLite shows that lightweight predicted gaze can replace hardware-measured gaze in gaze-guided MLLM video cropping without significant downstream description-quality loss. The conclusion is bounded by untested domain shift, human evaluation, deployment metrics, and controlled comparisons with prior predictors.
- Conclusion: Predicted-gaze crops matched ground-truth-gaze crops in downstream description quality across two MLLMs, three automated metrics, and two LLM judges, with equivalence confirmed in ten of ten cases.The result demonstrates substitution of lightweight model predictions for hardware-measured gaze in the gaze-guided MLLM video-cropping pipeline.
- Limitations: EgoGazeLite was evaluated in-distribution across seven Ego-Exo4D domains, leaving robustness to unseen activities and domain shift untested.A matched AAE/AUC comparison with Huang et al. and Lai et al. was also not performed because the models used different training data.
- Limitations: Description quality relied on automated metrics and LLM judges rather than human raters, while the study measured an iPhone 15 Pro phone-side pipeline and cropping-induced pixel-area reduction.End-to-end application latency, upload bandwidth, and MLLM-side visual token count were not directly measured.
- Future work: Future work should compare predictors under identical training splits, evaluate leave-one-domain-out transfer across all eight Ego-Exo4D domains, and isolate architectural components.These experiments would clarify efficiency–accuracy tradeoffs and transfer under domain shift.
- Conclusion: If the substitution result generalizes beyond the tested datasets, model families, and task domains, gaze-conditioned egocentric video understanding could extend to first-person cameras without dedicated eye-tracking hardware.This would broaden access beyond tracker-equipped wearables, conditional on results holding outside the evaluated settings.