Source-linked AI summary
Perception Test 2026: Challenge Summary and Extension to City-scale Audio-Visual Reasoning
Fedor Kitashov, João Carreira, Shiry Ginosar, Dima Damen, Andrew Zisserman, Viorica Pătrăucean
TL;DR
The challenge addresses spatial understanding, where frontier models’ general perception still falls short of average humans. It evaluates four tracks spanning the original Perception Test and new city-scale benchmarks, and reports improved results on the original tracks and near-perfect best-entry scores on the new tracks. The authors conclude that city-scale reasoning can be solved with expensive agentic systems but remains difficult for standalone multimodal models.
Problem
Frontier models’ general perception still falls short of average humans, especially on spatial understanding tasks.
Source: §1 · p. 1Method
The challenge evaluates four tracks using the original Perception Test benchmark and city-scale KilometerVision and KilometerAudio benchmarks, and summarizes winning solutions.
Source: §1 · p. 1Results
0.864 on KilometerAudio and 0.905 on KilometerVision were the best-entry scores, while unified multiple-choice video QA improved from 0.812 in 2025 to 0.914 top-1 accuracy in 2026.
Source: §4 · p. 4Takeaways & Limitations
The authors conclude that complex city-scale spatial and audio-visual reasoning can be solved with expensive agentic pipelines, while a single multimodal model trails winners on the city-scale tracks.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Continuing the Perception Test challenge series, we organised the fourth edition as a workshop at the European Conference on Computer Vision (ECCV) 2026 in Malm\"o, Sweden. This edition focused on spatial intelligence and featured four different tracks: unified multiple-choice videoQA and grounded videoQA from the original Perception Test benchmark, alongside two new tracks based on city-scale walking-tour videos (KilometerAudio and KilometerVision). In this report, we describe the new benchmarks used for the city-scale tracks and summarise the winning solutions across all tracks, including a generalist model that competed across all tracks with satisfactory performance. The winning solutions in the newly added city-scale tracks demonstrated that complex spatial and multimodal reasoning can be solved by expensive agentic pipelines, but remains difficult for multimodal models used standalone.
1. Introduction
The challenge probes spatial intelligence across environment scales, motivated by frontier models’ shortfalls in general perception, especially spatial understanding. It combines two original Perception Test tracks with two city-scale tracks based on walking-tour videos.
1. Introduction
- Frontier models’ general perception still falls short of average humans, especially on spatial understanding tasks.The paper contrasts this with models’ demonstrated capabilities in mathematics, plasma control, and software writing.Source: §1 · p. 1
- The challenge evaluates small-scale tabletop spatial capabilities with the original Perception Test benchmark and large-scale spatial understanding with KilometerVision and a new audio-visual benchmark.The report describes the new city-scale benchmarks and winning solutions across the tracks.Source: §1 · p. 1
- Four tracks comprise Unified VideoQA and Grounded VideoQA on original Perception Test videos, plus KilometerVision and KilometerAudio on publicly available YouTube walking-tour videos.The walking-tour videos were annotated specifically for the latter tracks.Source: §1 · p. 1
- The challenge awarded track winners and runners-up, plus a best generalist award for an entry using the same backbone across all tracks.The challenge offered 22K EUR in total prizes across tracks.Source: §1 · pp. 1–2
2. KilometerVision Benchmark
KilometerVision probes city-scale spatial capabilities using real-world walking-tour videos and multiple-choice video questions. Its configurations include text options, image options, or a reference image in the question.
2. KilometerVision Benchmark
- KilometerVision uses real-world YouTube walking-tour videos up to 10 minutes long, corresponding to about 1 km of walking.The benchmark probes city-scale spatial capabilities in frontier models.Source: §2 · p. 2
- Its multiple-choice videoQA configurations use five text options, five image options such as route maps, or a reference image in the question.The reference-image configuration can ask whether a landmark appears in the video and estimate its distance from the final camera position.Source: §2 · p. 2
- The seven question types are landmark recognition, distance to landmark, compass, loop closure, route summary, map trace, and Euclidean distance.Map-trace options may include or omit text labels.Source: §2 · p. 2
3. KilometerAudio Benchmark
KilometerAudio is introduced to test audio-visual reasoning in long urban walking-tour videos, where sounds may come from outside the camera’s view. Its questions focus on audio events and require both audio and visual information.
3. KilometerAudio Benchmark
- KilometerAudio tests audio-visual reasoning in very long walking-tour videos, including cases where audio events occur out of the camera’s view.Public YouTube videos retain naturally captured sounds and are not edited to add irrelevant sounds such as overlay music.Source: §3 · p. 2
- Figure 2’s landmark example asks whether a reference image shows a location in the video and, if so, the straight-line distance to the final frame’s camera location.The KilometerAudio split table is identified as Table 1, but its contents are not supplied here.Source: Figure 2 · p. 3, Table 1 · p. 4
- The test set contains 500 five-option questions over 218 public YouTube walking-tour videos lasting 1–3 hours.The benchmark focuses exclusively on audio events, unlike Video-MME Long, whose videos last up to one hour and whose questions only partly focus on audio.Source: §3 · p. 3
- Human raters identified recognizable sounds and wrote questions requiring both modalities, covering spatial sound sources, audio-event localization or timing, counting, songs, and speech.Raters also supplied answers and plausible difficult negative options; the validation examples require reasoning over the audio track.Source: §3 · p. 3, Figure 3 · p. 3
- All test questions are new, and only 3 of 218 test videos were used in previous editions; no training data are provided, so the track is zero-shot.Source: §3 · p. 3
4. Overall challenge summary
The 2026 challenge received 1,254 test submissions from 120 teams. Results improved on both original Perception Test tracks, while the best entries in the two new city-scale tracks approached perfect performance.
4. Overall challenge summary
- The paper reports 1,254 test submissions; Figure 6 shows the best test score of each track over the test phase, while servers for two new tracks went public later during August 2026.Source: §4 · p. 4
- 0.812 in 2025 to 0.914 top-1 accuracy in 2026 marked unified multiple-choice video QA improvement; grounded video QA rose from 0.497 to 0.664 HOTA.These tracks use videos from the original Perception Test benchmark.Source: §4 · p. 4
- 0.864 on KilometerAudio and 0.905 on KilometerVision were achieved by the best-performing entries, while prize-eligible submissions reached 0.782 and 0.753, respectively.The best-performing entry in each new track was not eligible for prizes.Source: §4 · p. 4
5. Challenge Tracks, Results, Awards
The 2026 challenge reports setups, results, and awarded methods across tracks using the original Perception Test and city-scale walking-tour videos. It also presents a single-model entry that competed across all four tracks.
5.1. Unified multiple-choice video QA
- Njust-KMG won the multiple-choice video QA test set with 91.4% using a question-type-aware weighted ensemble, while runner-up F423 reached 91.0% with strict answer validation and at most one re-query.Source: §5.1 · pp. 4–5
- 66 of 141 questions were quasi-solved in Unified VideoQA, versus 36 in 2025; only two scored 50% or lower, both in Physics.Physics accounted for 8 of the 10 hardest questions; the remaining two were in Spatial reasoning and Abstraction.Source: §5.1 · p. 5
5.2. Grounded video QA
- HOTA rose from 0.497 to 0.664 in Grounded VideoQA, alongside detection gains from 0.479 to 0.654 and association gains from 0.524 to 0.677.The winning pipeline paired a VLM with a promptable tracker.Source: §5.2 · p. 6, Table 3 · p. 6, Figure 9 · p. 7
5.3. KilometerVision
- 0.905: CloudAI-evolve achieved KilometerVision’s best overall score with an agentic pipeline, though it was not prize-eligible.The test set comprised 986 five-option questions on 54 walking-tour video segments; the strategy used validation data and about 10% of unlabeled test questions.Source: §5.3 · pp. 6–7, Table 5 · p. 7
5.4. KilometerAudio
- 0.864: CloudAI-evolve recorded KilometerAudio’s best overall score using an agentic pipeline with signal-processing tools and Gemini models.The winning prize-eligible entry, IUCV, proposed AVRA, a training-free audio-visual reasoning agent.Source: Figure 10 · p. 8, §5.4 · p. 8
5.5. Best Generalist award
- Wind_Rain_Tower competed across all four tracks with one training-free approach, placing 5th, 13th, 9th, and 22nd, respectively.Qwen3.8-Max made the final answers and box decisions; the approach trailed the per-track winners but was described as satisfactory across tracks.Source: §5.5 · p. 8
6. Discussion
Three tracks almost saturated, while Grounded VideoQA remained the exception. The best city-scale results came from expensive, slow agentic pipelines, whereas standalone multimodal models remained behind.
6. Discussion
- The three tracks other than Grounded VideoQA exceeded 80% accuracy; the two city-scale tracks were run for the first time in 2026.Source: §6 · pp. 8–9
- The challenge drew 1,254 test submissions from 120 teams across four tracks.Source: §6 · p. 8
- Almost all top solutions were zero-shot agentic VLM pipelines, often using ensembling or self-consistency; only the Unified VideoQA winner fine-tuned on Perception Test training data.The best city-scale methods used expensive, slow tool use, audio indexing, or geometric reconstruction, while the single-model generalist trailed on those tracks.Source: §6 · p. 9