Source-linked AI summary
EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering
Sheng Zhou, Junbin Xiao, Qingyun Li, Yicong Li, Xun Yang, Dan Guo, Meng Wang, Tat-Seng Chua, Angela Yao
TL;DR
Existing VideoQA and scene-text QA benchmarks provide limited evidence about real-time egocentric assistance because they emphasize visual understanding, clean images, or explicitly localized text. EgoTextVQA constructs a timestamped benchmark of scene-text-aware questions from outdoor and indoor ego-view videos and evaluates multimodal models. All evaluated models struggle, with Gemini 1.5 Pro reaching only around 33% accuracy, while analyses identify temporal grounding, multi-frame reasoning, higher resolution, and auxiliary scene-text inputs as useful directions.
Problem
Existing benchmarks do not adequately evaluate real user needs requiring scene-text reasoning in dynamic egocentric videos, often assuming clean images or explicitly localized text.
Method
EgoTextVQA constructs a timestamped benchmark from 1.5K ego-view videos and 7K scene-text-aware questions covering outdoor driving and indoor housekeeping.
Results
All 10 evaluated multimodal models struggle on EgoTextVQA, with Gemini 1.5 Pro achieving around 33% accuracy.
Takeaways & Limitations
Precise temporal grounding, multi-frame reasoning, high-resolution inputs, and auxiliary scene-text inputs are identified as promising directions for improving egocentric scene-text QA.
Takeaways & Limitations
Answer diversity limits human performance, and the authors plan to enrich the ground-truth answers for each question.
Abstract
from arXiv · showhide
We introduce EgoTextVQA, a novel and rigorously constructed benchmark for egocentric QA assistance involving scene text. EgoTextVQA contains 1.5K ego-view videos and 7K scene-text aware questions that reflect real user needs in outdoor driving and indoor house-keeping activities. The questions are designed to elicit identification and reasoning on scene text in an egocentric and dynamic environment. With EgoTextVQA, we comprehensively evaluate 10 prominent multimodal large language models. Currently, all models struggle, and the best results (Gemini 1.5 Pro) are around 33\% accuracy, highlighting the severe deficiency of these techniques in egocentric QA assistance. Our further investigations suggest that precise temporal grounding and multi-frame reasoning, along with high resolution and auxiliary scene-text inputs, are key for better performance. With thorough analyses and heuristic suggestions, we hope EgoTextVQA can serve as a solid testbed for research in egocentric scene-text QA assistance. Our dataset is released at: https://github.com/zhousheng97/EgoTextVQA.
1. Introduction
EgoTextVQA addresses the gap between practical egocentric assistance and existing VideoQA and scene-text benchmarks by introducing questions requiring scene-text-aware reasoning in dynamic first-person videos. Its benchmark reveals that current multimodal models struggle substantially, while temporal grounding, multi-frame reasoning, resolution, and auxiliary text inputs are promising improvement directions.
- Motivation: Existing VideoQA datasets emphasize visual understanding rather than questions reflecting real user needs in assistive applications.Related egocentric datasets focus on episodic memory or how-to demonstrations and neglect scene text.
- Motivation: Scene-text QA datasets commonly assume clean images and questions that explicitly identify text regions, limiting their practical assistive value.These settings primarily evaluate OCR under ideal imaging conditions rather than dynamic egocentric assistance.
- Benchmark: EgoTextVQA contains 1.5K ego-view videos and 7K scene-text-aware questions from outdoor driving and indoor housekeeping, with answers based only on content before each question timestamp.Questions reflect user needs without necessarily pinpointing scene text, while timestamped access simulates real-time video QA.
- Benchmark Challenges: The benchmark requires reasoning about user intentions, multiple frames, and the user’s current state to produce meaningful assistance.These requirements extend beyond recognizing isolated scene text or understanding a single frame.
- Evaluation: 10 prominent multimodal models are evaluated, but all struggle; Gemini 1.5 Pro achieves only around 33% accuracy.The benchmark therefore exposes severe deficiencies in egocentric scene-text QA assistance.
2. Related Work
Prior scene-text benchmarks generally use images or text-focused videos with clean visual attention and simple identification questions, while EgoTextVQA targets real-life egocentric assistance. Recent MLLMs improve image scene-text QA through higher resolution, but video comprehension remains less explored.
- Scene-Text VQA Benchmarks: Existing scene-text datasets often provide high-quality images or text-rich videos where questions directly point to well-focused scene text.Such designs limit practical assistance because they mainly test text identification; RoadTextVQA also focuses only on road driving.
- EgoTextVQA: EgoTextVQA broadens the setting to diverse daily scenarios with questions reflecting real user needs without visual attention focused on scene text.Its egocentric perspective is intended to advance real-life QA assistance.
- Multimodal Models: Higher-resolution image inputs have improved MLLM performance on image scene-text QA, but these models remain limited to image-level input while video comprehension is largely unexplored.Recent video-capable models address this modality gap, motivating evaluation in dynamic egocentric scenes.
3. EgoTextVQA Dataset
EgoTextVQA is constructed from filtered ego-view videos and MLLM-generated, human-refined questions designed around realistic, scene-text-aware assistance. The resulting benchmark contains 7,064 QA pairs across 1,507 videos, spanning outdoor driving and indoor housekeeping with timestamped, text-related questions.
- Raw Video Filtering: The source videos come from public ego-view datasets covering outdoor driving and indoor housekeeping, with filtering for scene-text-rich content and unsuitable footage.The construction process uses existing text-rich video sequences and removes videos that fail scene-text or quality criteria.
- QA Generation: Questions are generated from sampled video segments and designed to be goal-oriented, scene-text-aware, colloquial, first-person, and challenging beyond a single image.The protocol does not require every answer to transcribe scene text exactly.
- Manual Participation: Nine trained students manually review, amend, refine, and filter generated QA pairs, with additional annotation for timestamp-shifted real-time questions.About 10% of suitable time-sensitive questions are copied to different timestamps and re-answered.
- Dataset Analysis: Table 3 evaluates models using uniformly sampled pre-timestamp frames at low resolutions of 640×360 and 960×540, alongside a human study on 30% of questions.The table highlights the best and second-best results under the stated frame and resolution conditions.
- Dataset Comparison: About 45% of answers and 52% of questions reference exact scene text, and each real-time question is tied to a timestamp whose preceding video content determines the answer.This distinguishes EgoTextVQA from benchmarks that answer from the entire video or focus only on explicitly identified text.
4. Experiments
The evaluation measures ten MLLMs on EgoTextVQA using semantic answer similarity, showing broad difficulty across outdoor and indoor scenarios. Models perform weakly overall, with real-time QA and fine-grained scene-text reasoning remaining especially challenging.
- Evaluation Setup: Accuracy and Score are computed by GPT-4o mini from semantic similarity between predicted and ground-truth answers.Accuracy is the percentage of evaluator “yes” answers, while Score ranges from 0 to 5.
- Evaluation Setup: Ten MLLMs are evaluated, including three closed-source APIs and seven open-source video-capable models.The evaluated models include GPT-4o, Gemini 1.5 Flash, Gemini 1.5 Pro, and seven open-source systems.
- Main Results: 33–34% accuracy is achieved by Gemini 1.5 Pro on both EgoTextVQA-Outdoor and EgoTextVQA-Indoor, while all models struggle overall.Gemini exceeds the best open-source models by approximately 5% outdoors and 9% indoors.
- Error Analysis: The benchmark exposes persistent scene-text perception difficulty in complex, dynamic, egocentric environments, with the best outdoor system trailing humans by approximately 10%.The cited results characterize a gap between advanced MLLMs and human performance on the benchmark.
- Main Results: 20.2% is the highest accuracy on the real-time QA subset, compared with 33.4% on the full outdoor set.MiniCPM-V 2.6 ranks among the top two open-source models on real-time QA despite performing worst on the full set, largely by identifying unanswerable questions.
- Error Analysis: Indoor models struggle especially on Kitchen and Shopping questions, often giving vague locations instead of specific shelf or object positions.Human performance is also affected by low resolution, motion blur, occlusion, long videos, external knowledge needs, and answer diversity.
5. Heuristic Solution Investigations
The heuristic investigations test temporal inputs, frame resolution, scene-text enhancement, and modality combinations. Results indicate that performance depends on how frames are selected and represented, with combined video and scene-text inputs producing the strongest explored results.
- Temporal Inputs: Single-frame image understanding can sometimes solve EgoTextVQA, but almost all models lose performance relative to uniformly sampled video frames.Qwen2-VL is an exception on EgoTextVQA-Outdoor, where a single frame improves its performance.
- Temporal Inputs: QA Frames consistently improve EgoTextVQA-Indoor but have inconsistent effects outdoors, including gains of 3.7% for Qwen2-VL and 2.1% for InternVL2-8B.MiniCPM-V 2.6 and Gemini 1.5 Pro instead decrease by 2.1% and 7.1%, respectively, on the outdoor comparison.
- Temporal Inputs: Question-timestamp-aware backward sampling at 1 fps improves Qwen2-VL outdoors even with fewer frames, whereas Gemini 1.5 Pro benefits from more frames.The strategy samples backward from the question timestamp to reflect the user’s recent visual context.
- Resolution: 18.6–31.0% improvements occur for GPT-4o, Qwen2-VL, InternVL2-8B, and MiniCPM-V 2.6 when frame resolution increases.At low resolution, three QA Frames are comparable to uniform video sampling, but higher-resolution inputs improve performance.
- Scene-Text Inputs: Scene-text super-resolution magnifies detected text regions to 380 × 128 while varying their projected scale and area ratio.The investigation uses Azure OCR for detection and DiffTSR for scene-text super-resolution.
- Modality Inputs: Adding scene text to video and question inputs yields the highest results in the explorations, including 52.9% for GPT-4o outdoors and 39.5% for Gemini indoors.Video input is more important than image input for almost all models, while auxiliary scene-text input further boosts performance.
6. Conclusion
EgoTextVQA is presented as a benchmark for real-life egocentric scene-text QA assistance across outdoor driving and indoor housekeeping. Its evaluations show that strong existing scene-text models still struggle, while heuristic analyses provide improvement directions.
- Conclusion: EgoTextVQA covers real user needs in outdoor driving and indoor housekeeping, often requiring visual attention beyond explicitly indicated scene text.The benchmark includes classified visual scenarios and question types for analyzing egocentric assistance.
- Conclusion: Existing scene-text-capable models struggle on EgoTextVQA, motivating further work on egocentric video QA assistance.The conclusion frames the benchmark as complementary to existing VideoQA research.
- Conclusion: The paper uses heuristic solution investigations to provide insights for improving performance on egocentric scene-text QA.The conclusion presents these analyses as directions for future research rather than as a complete solution.
A. EgoTextVQA Dataset
Manual participation substantially filters and revises automatically generated QA pairs to improve their validity and alignment with real user needs. The process addresses hallucinations, scene-text errors, irrelevance, unnatural wording, and unrealistic questions.
- Manual Participation: About 70% of GPT-4o-generated QA pairs are deleted and 30% of the remaining pairs are revised after manual participation.Manual correction targets hallucinated answers, irrelevant or incorrect scene text, non-colloquial phrasing, and questions that do not reflect user needs.
- Manual Participation: Automatic generation produces hallucinated answers that annotators cannot confirm from the visual environment.These answers are unseen in the videos and therefore cannot be grounded in the depicted scene.
- Manual Participation: Generated questions may fail to incorporate scene-text understanding or contain incorrect scene text in the questions or answers.These issues directly undermine the intended scene-text QA task.
- Manual Participation: Generated questions can be mechanical, unnatural, or disconnected from ordinary daily user needs.Manual review is used to improve colloquial quality and practical relevance.
B.1. Model Details
The paper evaluates prominent multimodal language models and analyzes design factors associated with stronger EgoTextVQA performance.
- Qwen2-VL uses a ViT-675M visual encoder, Qwen2-7B language model, MLP projector, dynamic resolution, and multimodal rotary position embeddings.
- LLaVA-NeXT-Video combines SigLIP-SO400M, Qwen2, a two-layer MLP projector, and AnyRes for video reasoning.
- 1.2% and 5.8% performance gains result from increasing video frames and resolution for Qwen2-VL.
- 7% higher performance is observed for InternVL2-26B compared with InternVL2-8B.
B.3. Heuristic Solution Investigations
The heuristic studies examine timestamp-aware sampling, model design, and auxiliary inputs to improve EgoTextVQA performance, while showing that gains can depend on the dataset.
- Timestamp-Aware Sampling: Timestamp-aware fixed-duration sampling achieves comparable or superior performance to whole-video uniform sampling for Qwen2-VL and Gemini 1.5 Pro.
- MLLM Design: High-resolution and long-video-capable visual encoders, stronger language-model backbones, and large-scale OCR training data are identified as factors associated with stronger performance.
- Combination of Heuristic Strategies: Progressively combining heuristic strategies produces cumulative gains on EgoTextVQA-Outdoor but unstable improvements on EgoTextVQA-Indoor.
- Human Evaluation: Human performance improves when temporal grounding is reduced by providing question frames, but still trails GPT-4o.
- Combination of Heuristic Strategies: Table 10 studies timestamp-aware sampling, additional scene-text input, and 1.25× high-resolution scene text using 30% of the data.
B.5. Case Analysis
Case analyses show failures in temporal grounding, scene-text interpretation, state tracking, and multi-step reasoning across outdoor and indoor examples.
- EgoTextVQA-Outdoor: Models often fail to identify the target at the queried timestamp or infer the user’s intended answer in outdoor cases.
- EgoTextVQA-Indoor: All models fail to recover the shopping total of 46.85 after checkout, requiring aggregation across changing cashier-display prices.
- EgoTextVQA-Outdoor: Only Gemini 1.5 Pro correctly answers the medical-assistance question with Vijaya hospital in the illustrated example.
- EgoTextVQA-Outdoor: Different timestamps can change the correct spatial relation, yet models repeatedly answer that the fotografiska building is to the right when the ground truth is straight ahead.
- EgoTextVQA-Outdoor: Models struggle to track real-time object states and actions, such as a rule sheet moving into another player’s hand.
- Human Evaluation: Human and GPT-4o mini judgments are compared using sampled outdoor question-answer pairs and shared evaluation criteria.