PaperlayerView exact revision on arXiv

Source-linked AI summary

Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering

Yura Choi, Roy Miles, Rolandos Alexandros Potamias, Ismail Elezi, Jiankang Deng, Stefanos Zafeiriou

arXiv:2603.12533v2cs.CV

TL;DR

MLLMs struggle with gesture-grounded egocentric questions because gesture-rich data are scarce and pointing intent is difficult to infer. The paper introduces EGOPOINTVQA and HINT, which adds hand-keypoint-derived tokens to model inputs, and reports state-of-the-art performance across gesture-grounded tasks.

  • Problem

    MLLMs have limited gesture-rich egocentric video data and limited ability to infer fine-grained pointing intent for deictic questions.

  • Method

    EGOPOINTVQA provides gesture-grounded egocentric VQA data, while HINT interleaves tokens derived from 3D hand keypoints with visual and textual inputs.

  • Results

    HINT achieves state-of-the-art performance, outperforming open-source baselines by an average of 5.4%.

  • Takeaways & Limitations

    The benchmark and explicit hand-intent representation establish a foundation for evaluating and improving gesture-grounded egocentric question answering.

Abstract

from arXiv · show

Understanding and answering questions based on a user's pointing gesture is essential for next-generation egocentric AI assistants. However, current Multimodal Large Language Models (MLLMs) struggle with such tasks due to the lack of gesture-rich data and their limited ability to infer fine-grained pointing intent from egocentric video. To address this, we introduce EgoPointVQA, a dataset and benchmark for gesture-grounded egocentric question answering, comprising 4000 synthetic and 400 real-world videos across multiple deictic reasoning tasks. Built upon it, we further propose Hand Intent Tokens (HINT), which encodes tokens derived from 3D hand keypoints using an off-the-shelf reconstruction model and interleaves them with the model input to provide explicit spatial and temporal context for interpreting pointing intent. We show that our model outperforms others in different backbones and model sizes. In particular, HINT-14B achieves 68.1% accuracy, on average over 6 tasks, surpassing the state-of-the-art, InternVL3-14B, by 6.6%. To further facilitate the open research, we will release the code, model, and dataset. Project page: https://yuuraa.github.io/papers/choi2026egovqa

1. Introduction

The paper targets gesture-grounded egocentric VQA, where MLLMs struggle to connect deictic language with the objects indicated by pointing. It introduces EGOPOINTVQA and HINT to provide gesture-rich evaluation data and explicit hand-intent context.

  • Research gap: MLLMs struggle with gesture-grounded, region-specific questions because existing data contain limited natural pointing and egocentric interaction examples.Current architectures also lack mechanisms to explicitly connect hand motion and pointing direction with deictic language.
  • Dataset: EGOPOINTVQA contains 4,000 synthetic and 400 real egocentric videos for evaluating deictic questions across fine-grained spatial and temporal reasoning.Its benchmark includes 672 question–answer pairs over 300 real-world videos.
  • Method: HINT converts per-frame 3D hand keypoints into frame-aligned gesture tokens and interleaves them with visual and textual inputs.The tokens provide an explicit gesture stream for interpreting pointing intent.
  • Results: Training with gesture tokens improves deictic-question performance by 6.5% over standard fine-tuning and surpasses the baseline by 8.6%.The method is presented as state-of-the-art across gesture-grounded question-answering tasks.

2. Related Work

Prior egocentric VQA benchmarks test reasoning about actions and environments, while this work focuses on natural pointing gestures as an explicit cue for deictic visual grounding.

  • Egocentric VQA: Existing egocentric VQA benchmarks evaluate complex reasoning about the wearer’s actions and environment.Examples include EgoThink and VidEgo-Think, which use first-person questions.
  • Task scope: EGOPOINTVQA decomposes gesture-grounded questions into reference, counting, spatial, temporal, attribute, and feedback tasks.These categories concern properties of pointed objects and require visual grounding of deictic references.
  • Positioning: Unlike approaches relying on artificial prompts, HINT conditions an MLLM on natural pointing gestures by interleaving continuous tokens derived from 3D hand keypoints.The approach complements prior work using gaze to align language with visual intent.

3. EGOPOINTVQA Dataset

EGOPOINTVQA combines synthetic and real egocentric videos with automatically generated deictic multiple-choice questions about pointed objects. Its taxonomy spans six reasoning categories, while the collection and generation pipelines provide controlled scenes and structured annotations.

  • Dataset overview: EGOPOINTVQA pairs real-world and synthetic egocentric videos with multiple-choice questions about specific visible regions and objects.The dataset is designed specifically for pointing gesture-based question answering.
  • Task formulation: Deictic questions use first-person expressions whose referents cannot be determined without visual and gestural context.Answering requires spatial-temporal hand–object alignment, linguistic grounding, and reasoning about object properties and scene context.
  • Video collection: Synthetic videos provide controlled object placement, viewpoint, and gesture timing, while real videos capture natural behavioral variability.Training is predominantly synthetic with some real videos, whereas testing uses real-world footage.
  • Synthetic generation: 4,000 synthetic videos are generated in AI2-THOR from diverse indoor scenes, with pointing animations aligned to selected objects using inverse kinematics.The clips are rendered as 3–5 second videos at 30 frames per second.
  • Question generation: The question-generation pipeline extracts dense video descriptions and metadata, creates structured questions about target objects, and converts them into first-person deictic expressions.Real evaluation videos receive manual verification and refinement.
  • Statistics: The dataset contains 4,000 synthetic videos, 400 real videos, and 18,745 question-answer pairs.The training set includes whole synthetic videos plus 100 real videos and 640 QA pairs; the test set contains 300 real videos and 672 QA pairs.

4. Hand Intent Tokens

HINT augments the visual stream with frame-aligned hand-intent tokens derived from 3D hand keypoints, enabling explicit modeling of pointing location, movement, and temporal context. These tokens are interleaved with visual and language inputs so the MLLM can ground deictic answers in the user’s gesture.

  • Architecture: HINT processes video through parallel visual and hand-intent streams before combining their tokens for the language model.The visual stream produces frame embeddings, while the hand-intent stream converts 3D hand pose features into gesture tokens.
  • Hand Intent Token Extraction: 21 3D hand keypoints per frame are projected by a lightweight adapter into one hand-intent token representing the hand’s posture.WiLoR estimates camera-space coordinates for the keypoints, which serve as adapter input.
  • Frame-Keypoint Interleaving: HINT interleaves hand-intent tokens with corresponding visual tokens, providing the LLM with an explicit gesture stream alongside video and text.This construction associates hand motion and pointing direction with deictic questions such as “this one.”
  • Hand Intent Token Extraction: The adapter omits a hand-intent token when hand-detection confidence is below τ, allowing intermittent hand visibility to be handled naturally.The implementation uses a confidence threshold of τ = 0.5 for inserting keypoint tokens.
  • Frame-Keypoint Interleaving: Because HINT tokens precede later predictions in the sequence, answer generation is conditioned on temporally anchored hand signals.The conditional formulation explicitly includes preceding HINT tokens with preceding instructions and answer tokens.

5. Experiments

Experiments show that EGOPOINTVQA remains challenging for existing MLLMs, while HINT consistently improves gesture-grounded performance with low computational overhead. Ablations indicate that explicit hand-intent modeling, mixed training data, appropriate confidence filtering, and temporal context are important.

  • Main Results: All evaluated models average below 70% accuracy on EGOPOINTVQA, and GPT-5 shows a dramatic drop from Reference to Temporal tasks.
  • Main Results: HINT consistently improves open-source backbones across task categories, with InternVL3-14B gaining 10.7 percentage points in Reference accuracy.InternVL3-14B improves from 63.1% to 73.8%.
  • Main Results: HINT improves LLaVA-OneVision by an average of 4.5 percentage points across six tasks.
  • Efficiency: HINT tokens increase InternVL3-8B inference time from 2.58s to 2.84s while contributing less than 1% of total LLM tokens.The measurement uses 32-frame inputs and a confidence threshold of τ = 0.5.
  • Ablation Study: Combining supervised fine-tuning with HINT reaches 75.0% Reference accuracy, compared with 68.5% from supervised fine-tuning alone.The ablation indicates that gesture-question exposure alone provides only a small benefit, whereas adding hand-intent processing yields the larger gain.
  • Ablation Study: Mixing synthetic and real videos yields the best training result, while a confidence threshold of τ = 0.5 best balances gesture retention and noise filtering.Low thresholds introduce noise, whereas high thresholds discard valid gestures.
  • Ablation Study: Uniformly sampled temporal inputs outperform oracle keyframes, showing that pointing understanding requires context beyond isolated gesture frames.The effect is especially pronounced for the Reference task.

6. Conclusion

The paper introduces EGOPOINTVQA for gesture-grounded egocentric video question answering and HINT for encoding hand intent. HINT achieves state-of-the-art performance while outperforming open-source baselines by an average of 5.4%.

  • EGOPOINTVQA contains 4,000 synthetic and 400 real-world videos for fine-grained spatial and temporal reasoning from pointing gestures.
  • HINT encodes 3D hand keypoints as tokens interleaved with visual and textual inputs to interpret referential intent.
  • HINT achieves state-of-the-art performance, outperforming open-source baselines by an average of 5.4%.

A.1. HINT Training Details

HINT training updates only the LoRA and keypoint adapters while keeping the backbone frozen, using AdamW, cosine decay, warm-up, and fixed 32-frame inputs.

  • Hyperparameters are selected through grid search over LoRA rank-scaling pairs and learning rates using best validation performance for each backbone.The tested rank-scaling pairs are {(8, 16), (16, 32), (32, 64), (64, 128)}, with learning rates {2e-7, 1e-5, 2e-5}.
  • Training updates only LoRA and keypoint adapters while freezing the remaining backbone parameters.Optimization uses AdamW, cosine learning-rate scheduling, and a 0.03 linear warm-up ratio.
  • Input videos are uniformly sampled to 32 frames, resized per backbone settings, and normalized using default preprocessing.

A.2. Evaluation Protocol

Evaluation uses multiple-choice accuracy with normalized option-letter extraction, while general video understanding is tested on established benchmarks without target-task fine-tuning.

  • Evaluation Protocol: Models are evaluated by whether the extracted option letter exactly matches the ground-truth answer.The protocol accepts several formatted answer patterns and marks unmatched outputs invalid.
  • Generalization Evaluation: HINT is compared with InternVL3-8B on Video-MME, MVBench, EgoSchema, and EgoBlind using 32 frames without fine-tuning those target benchmarks.The evaluation tests whether gesture-aware fine-tuning preserves general video understanding capabilities.
  • Evaluation Protocol: All reported baselines and HINT variants produced valid option letters for every prediction, so accuracies require no manual corrections.

B.2. Human Performance

Human participants perform near ceiling on EGOPOINTVQA, while the best-performing model remains substantially lower. Removing visible hand gestures causes a large performance degradation across tasks.

  • 95.9% total average accuracy was achieved by five human participants evaluating the full 672-question test set.The result indicates that the questions are clear and easy for humans.
  • 68.1% was the best-performing model accuracy, leaving a substantial gap to human performance.
  • Removing the pointing hand caused a large performance degradation across all tasks.The static-video variant kept the scene and camera motion unchanged while removing the wearer’s hand.

B.4. Dataset Bias Analysis

Text-only baselines perform near random chance, indicating that EGOPOINTVQA requires visual grounding rather than relying on textual or option-distribution shortcuts.

  • Baseline MLLMs exhibit saliency bias and temporal confusion when resolving pointed references.They may select a prominent or central object, or confuse the order of sequentially pointed objects.
  • Near-random performance from the Blind and Choices-only baselines confirms that EGOPOINTVQA requires visual grounding.Blind receives only the question text, whereas Choices-only receives only the answer options.

B.5. Failure Analysis

Failure analysis contrasts baseline errors from saliency and temporal confusion with HINT errors caused by unreliable hand keypoints and rapid viewpoint drift.

  • Failure Analysis: Baseline MLLMs often select visually prominent or centrally located objects instead of the pointed referent.This saliency or center bias is especially observed in cluttered scenes.
  • Failure Analysis: Baseline models frequently confuse the temporal order of objects pointed at sequentially.
  • Failure Analysis: HINT mitigates baseline spatial and temporal errors by providing frame-aligned 3D hand geometry.
  • Failure Analysis: HINT can fail when motion blur or partial occlusion makes 3D hand reconstruction noisy, or when viewpoint drift is rapid.

C.2. Dataset Statistics

EGOPOINTVQA combines large-scale synthetic data with real-world indoor and outdoor examples, while its generation pipeline constructs gesture-grounded questions from structured scene information.

  • Dataset Statistics: 4,000 synthetic videos form the large-scale training set, complemented by real-world data for domain adaptation and testing.
  • Dataset Statistics: Real-world clips generally contain higher object density than synthetic clips, increasing the challenge of grounding pointed targets.Reported averages are 22.5 and 16.5 objects per real-world video versus 11.6 per synthetic video.
  • Dataset Statistics: Target objects have a diverse spatial distribution across frame coordinates rather than being confined to the center.The distribution requires models to use pointing gestures instead of relying only on center priors.
  • Dataset Statistics: The dataset includes qualitative examples from indoor and outdoor real-world scenes, alongside synthetic training examples generated with AI2-THOR.
  • Dataset Statistics: The three-stage generation pipeline extracts dense scene information, generates question-answer pairs, and rephrases them into natural deictic questions.InternVL3-78B handles scene extraction and QA generation, while GPT-4o performs linguistic refinement and quality control.
  • Dataset Statistics: Task-specific prompts generate reference, attribute, spatial, feedback, counting, and temporal questions.
  • Dataset Statistics: Negative choices are created from neighboring-object attributes or plausible but incorrect properties to make multiple-choice options challenging.
Loading 2603.12533v2…