Source-linked AI summary
Results of the 1st Asynchronous CASTLE Challenge at the Joint Egocentric Vision Workshop in Conjunction with CVPR 2026
Luca Rossetto, Werner Bailer, Cathal Gurrin, Graham Healy, Omar Shahbaz Khan, Stevan Rudinac, Klaus Schöffmann, Allie Tran
TL;DR
The challenge examined closed-form QA over long-term, multimodal egocentric data through a 185-question benchmark. Teams submitted evidence-retrieval, agentic, and knowledge-graph approaches, and the results suggest combining evaluated methods could be more effective. The ranked teams collectively answered 176 questions, while accuracy ranged from 0.351 to 0.584.
Problem
The challenge tested whether systems could answer 185 multiple-choice questions requiring reasoning across long-term, multimodal, multiview egocentric data.
Method
Teams used evidence-aware retrieval, source-selective agentic reasoning, hierarchical knowledge graphs, and staged search–verification–answer pipelines.
Results
176 of 185 questions were correctly answered by the ranked teams in aggregate, while individual accuracies ranged from 0.351 to 0.584.
Takeaways & Limitations
The result analysis suggests that combining the evaluated approaches might be more effective in future challenge editions.
Abstract
from arXiv · showhide
This report summarizes the contributions and results of the 1st Asynchronous CASTLE Challenge at the Joint Egocentric Vision Workshop in conjunction with CVPR 2026.
1. Introduction
The CASTLE Challenge introduced closed-form QA over long-term, multimodal egocentric data using the CASTLE 2024 dataset. Its 185-question benchmark tested reasoning across multimodal, multiview inputs.
- The pilot challenge used CASTLE 2024, a four-day collection of high-fidelity, multiperspective recordings in a controlled environment.
- The dataset contains over 600 hours of UHD video at 50 frames per second with synchronized audio from 10 participants performing everyday activities.
- Its multimodal recordings combine 10 egocentric and 5 static video streams with 6DoF IMU, GPS, biometric, gaze, and thermal data.
- 1.2. Task Definition: The benchmark comprised 185 four-choice questions requiring systems to reason across multimodal, multiview inputs.
- 1.2. Task Definition: Questions tested temporal reasoning, object tracking, visual counting, spatial awareness, activity recognition, and participant identification.
2. Team Contributions
The challenge attracted substantial participation and produced competitive but difficult results, with teams exploring evidence retrieval, agentic source selection, knowledge graphs, and staged verification. Across approaches, reported analyses emphasized structured evidence access and multimodal coverage.
- 43 registered participants produced 272 leaderboard submissions, while four teams submitted technical reports.
- Top-performing teams achieved 50%–58% accuracy, showing that long-context, closed-form QA over 600 hours of multiview data remained challenging.
- 2.1. Team WDL: Team WDL converted long-form QA into evidence retrieval and verification using Qwen, query-guided multimodal retrieval, question-type routing, LoRA adaptation, and self-consistency.
- 2.2. Team MARS: Team MARS dynamically selected which modality to query through an iterative agent loop operating over video summaries, transcripts, gaze, heart rate, photos, and thermal imagery.
- 2.2. Team MARS: Starting from 35% text-only accuracy, MARS gained 7% from official transcripts, 4% from contextual cues, and reached 57% with its full agentic loop.
- 2.3. Team TAHAKOM: Team TAHAKOM built a training-free hierarchical knowledge-graph retrieval framework mapping entities and relationships across long temporal contexts.
- 2.3. Team TAHAKOM: Adding Mention relationships improved TAHAKOM accuracy by 7%, while ClipSearch and ClipAnalyze added 4% by grounding visual queries in raw clips.
- 2.4. Team CuriosAI: CuriosAI’s SVA pipeline separated search, verification, and answering, using hybrid retrieval, adjacent camera windows, and evidence-priority fusion.
3. Result Analysis
The ranked teams answered 65–108 of 185 questions, achieving 0.351–0.584 accuracy and surpassing the 0.25 random baseline. Their complementary strengths enabled 176 questions to be solved collectively, leaving only 9 unanswered by every ranked team.
- 65–108 of 185 questions were answered correctly, yielding accuracies from 0.351 to 0.584.
- All teams substantially outperformed the random baseline of 0.25.
- 176 questions were answered correctly by the teams in aggregate, compared with 108 for the highest-scoring team.
- Only 9 questions were not correctly answered by any ranked team.
4. Conclusion
The report concludes that submitted approaches were promising, while suggesting that combining evaluated approaches could be more effective in future challenge editions.
- Submitted approaches were promising, but combining evaluated approaches might be more effective in future challenge editions.