Source-linked AI summary
Motion-Based Tokenization for Cross-Dataset Egocentric Gaze Modeling
Virmarie Maquiling, Zhuojiang Cai, Enkelejda Kasneci
TL;DR
Gaze modeling lacks a representation that is compact, comparable across datasets, and able to retain local motion structure. The paper introduces event-aligned angular-motion tokens and evaluates them against alternative representations with transfer and structural diagnostics, finding direction-specific advantages and showing that event construction materially shapes transfer.
Problem
Gaze representations must balance detail, noise, device dependence, comparability, and preservation of local motion structure across datasets.
Method
The paper formulates event-aligned, fixed-horizon angular displacement as an event-conditioned motion vocabulary and evaluates it with predictive, transfer, order, motif, and frozen-probe measures.
Results
Angular-motion tokens have lower target-domain regret than VQ tokens for AEA→RITW, while the reverse direction is inconclusive; EGTEA results show native events perform best into EGTEA and frame-span events fail as a source.
Takeaways & Limitations
Motion-token transfer depends on both vocabulary choice and the temporal and event interfaces used to construct gaze streams.
Takeaways & Limitations
The EGTEA comparison does not establish that one event interface is universally preferable because interfaces differ intrinsically in boundaries, temporal units, labels, lengths, and conditioning.
Abstract
from arXiv · showhide
Gaze is increasingly used as an input signal for vision and multimodal models, yet no consensus exists on how to represent it across datasets. Raw traces preserve detail but are noisy and device-dependent, while coarse event labels are easy to model but can discard local motion structure. We formulate event-aligned, fixed-horizon angular displacement as an interpretable, event-conditioned motion vocabulary and compare it with event-only, spatial, absolute-angle, learned vector-quantized, and continuous representations. To assess transfer alongside target predictability and token collapse, our evaluation combines next-token prediction with target-domain regret, low-order target references, paired bootstrap, order sensitivity, motif overlap, and frozen structural probes. In an event-aligned headset benchmark, angular-motion tokens have lower target-domain regret than frozen-codebook VQ tokens in one transfer direction, while the reverse direction is inconclusive. The probes reveal complementary representation properties, and event-only tokens show that low perplexity can retain little motion information. On a third egocentric dataset, a matched comparison of I-VT, native, and frame-span interfaces shows that event construction materially changes transfer: native events have the lowest regret into EGTEA, while frame-span events have zero motif overlap and fail severely as a source. Motion-based tokenization therefore provides a compact representation for event-aligned egocentric gaze streams, while the evaluation identifies how target predictability and event construction shape cross-dataset conclusions.
1. Introduction
The paper proposes event-aligned angular motion tokens to preserve local gaze dynamics while improving compactness and cross-dataset comparability. It evaluates this representation against alternative tokenizations using transfer and structural diagnostics.
- Raw gaze traces preserve detail but are noisy, device-dependent, and difficult to align, whereas coarse event labels can discard local motion structure.
- Event-aligned, fixed-horizon angular displacement is formulated as an interpretable, event-conditioned discrete vocabulary with reduced dependence on absolute gaze position under a shared interface.
- The study compares angular-motion tokens with event-only, spatial, absolute-angle, learned vector-quantized, and continuous representations across three egocentric datasets.
- The evaluation combines target-domain regret, low-order target references, paired bootstrap, order sensitivity, motif overlap, and frozen structural probes to assess transfer and retained structure.
- Angular-motion tokens outperform frozen-codebook VQ tokens in target-domain regret for AEA→RITW, while RITW→AEA remains inconclusive under paired bootstrap.
- The matched EGTEA comparison shows that event construction materially changes transfer, with native events lowest in regret and frame-span events showing zero motif overlap and severe source transfer failure.
2. Related work
Related work treats gaze representation as a choice that determines which spatial, temporal, and sequential properties remain measurable. This paper extends that perspective to reusable structure under dataset shift.
- Classical and scanpath methods represent gaze through movement properties, spatial-temporal letters, scanpath comparisons, latent states, recurrence, or subsequence frequencies.
- Discrete gaze tokenization improves alignment, sequence modeling, and motif analysis but trades fidelity for stability.
- The paper asks whether representations remain compact, predictable, and structurally reusable under cross-dataset shift rather than evaluating robustness only through end-task accuracy.
- Domain shift affects gaze models through changes in capture conditions, subjects, gaze ranges, and annotation procedures.
3. Method and Experimental Setup
The study trains shared next-token models on event-tokenized gaze streams and evaluates transfer with predictability, stability, sequence-order, motif, and frozen-probe diagnostics. It also tests tokenizer, temporal-scale, and event-interface sensitivity.
- Method and Experimental Setup: Cross-domain transfer trains on source-tokenized gaze events and evaluates both in-domain and held-out target-domain streams under a specified protocol.
- Datasets and interfaces: AEA and RITW form the primary transfer pair, while EGTEA provides an event-interface boundary check because its sensor and export format differ.
- Event construction: Discrete representations derive from event streams, including adaptive velocity-threshold events for AEA/RITW and degree-aware I-VT, native, and frame-span interfaces for EGTEA.
- Tokenizers: The primary AD tokenizer quantizes event-conditioned (∆yaw, ∆pitch) on a fixed 2D grid with explicit zero-motion tokens and configurable out-of-range handling.
- Model controls: All discrete models share architecture, splits, context length, filtering, optimizer family, and training budget, with CONT predicting continuous angular displacement using Gaussian negative log-likelihood.
- Transfer metrics: Degradation, perplexity, and bits/token are interpreted alongside target-domain regret because target predictability, alphabet size, and token rates differ across representations.
- Structural diagnostics: Order-manipulation controls compare real and shuffled training or test sequences to measure sequence-order dependence beyond token frequencies.
- Structural diagnostics: Frozen mean-pooled sequence-model states are probed for motion-state, motion-quadrant, and direction-reversal information.
4. Results
Across the AEA↔RITW benchmark, angular-displacement tokens combine competitive transfer with measurable sequential and motif structure, while results vary by direction, temporal scale, tokenizer design, and evaluation metric.
- Cross-domain transfer: 0.05±0.07 pooled degradation makes AD the lowest observed hand-designed tokenizer, ahead of ASM, SGE, AA, and SD.Direction-specific degradation is 0.01±0.08 for AEA→RITW and 0.09±0.04 for RITW→AEA.
- Cross-domain transfer: −0.964 bits/token paired raw-regret difference favors AD over VQD for AEA→RITW, whereas the 0.066 difference for RITW→AEA is inconclusive.The 95% confidence intervals are [−1.183, −0.756] and [−0.417, 0.616], respectively.
- Structural diagnostics: 0.40 ± 0.10 pooled order sensitivity and 0.493/0.449 bigram/trigram overlap show that AD retains sequential and local motif structure.AD exceeds AA, SGE, and SD on order sensitivity and has substantially higher motif overlap than all listed alternatives.
- Structural diagnostics: 0.625 ± 0.101 motion-quadrant decodability belongs to CONT, while AD leads discrete tokenizers at 0.481± 0.044; VQD and CONT instead lead generic motion detection.Motion detection reaches 0.960 ± 0.030 for VQD and 0.899 ± 0.091 for CONT, versus 0.605 ± 0.103 for AD.
- Temporal scale: 1.21 ± 0.07 OOD perplexity is best for AEA→RITW with 120 ms stride and context 32, while RITW→AEA favors 40 ms and context 64 at 1.022 ± 0.004.There is no single best temporal scale, and equal effective horizons can still produce different transfer.
- Temporal scale: 0.94 to 0.72 top-10 concentration reduction accompanies entropy growth from 2.43 to 3.29 when AEA stride increases from 40 ms to 120 ms.The diagnostics associate moderate smoothing with compression and aggregation effects that differ by dataset.
- Tokenizer sensitivity: −0.07 ± 0.10 mean degradation makes source-specific K = 128 VQD the strongest tested VQ policy, while pooled codebooks nearly eliminate motif overlap.Pooled-codebook bigram/trigram Jaccard values are 0.005 ± 0.003 and 0.003 ± 0.000.
5. Discussion
The discussion shows that transfer depends on discretization, temporal scale, and event interface, while practical use requires checking whether tokens preserve structure rather than merely predictability. The conclusions are bounded by the tested egocentric datasets, shared headset setting, and diagnostic scope.
- Representation choice: AD has lower target-domain regret than VQD for AEA→RITW, while the reverse direction is inconclusive.The comparison concerns complete representations rather than the quantizer alone.
- Representation choice: SO(3) quantization fragments local motion structure, whereas angle-only ablation recovers stronger transfer; VQD codebook usage does not ensure motif reuse.Bigram and trigram support nearly vanishes across AEA/RITW for SO(3), while pooled VQ centroids can destroy motif overlap.
- Temporal scale: Temporal scale changes transfer asymmetrically: AEA→RITW benefits from moderate smoothing, whereas RITW→AEA prefers finer tokens with longer context.The analyses suggest compression effects for AEA and weaker aggregation effects for RITW.
- Event interfaces: Native events have the lowest target-domain regret into EGTEA, while frame-span events have zero local motif overlap and fail severely as the source.Matched controls leave intrinsic differences in event boundaries, temporal units, labels, sequence lengths, and event conditioning.
- Practical implications: Motion-based tokens are compact inputs for temporal-attention models, but token streams should be checked for collapse, order sensitivity, and cross-domain motif mismatch.Poor transfer may reflect either a different gaze regime or an incompatible event interface.
- Limitations: The conclusions concern event-based egocentric gaze transfer under tested interfaces, with clearest comparative evidence from AEA and RITW's shared Project Aria device family.Larger balanced egocentric corpora are needed to separate dataset size, task regime, and sensor effects more cleanly.
- Privacy and ethical considerations: Discrete motion tokens are not anonymization: they reduce some spatial precision while preserving temporal structure that can encode behavioral signatures.Tokenization may reduce some risks of raw gaze traces while transforming others into sequence-level risks.
6. Conclusion
The paper studies which gaze abstraction remains useful for egocentric event-based transfer under dataset shift. It finds that motion tokenization can be compact and interpretable, but tokenizer design, event interfaces, and retained behavioral structure shape the transfer outcome.
- The study asks which gaze-tokenization abstraction remains useful for egocentric event-based transfer under dataset shift.
- AD has lower target-domain regret than VQD for AEA→RITW, while RITW→AEA remains inconclusive under paired bootstrap.
- Temporal and tokenizer-sensitivity analyses show that granularity, context length, grid design, event conditioning, and zero/out-of-range handling shape learnable structure.
- A matched EGTEA comparison shows that degree-aware I-VT, vendor-native, and frame-span interfaces expose different transfer and motif structure to the same tokenizer.Frame-span events fail severely when used as the source.
- Motion-based tokenization is a compact and interpretable option for compatible egocentric gaze streams when tokenizer design, event interfaces, and retained behavioral structure are explicitly considered.