MAP: A Benchmark on Multimodal Accessibility Planning for Real World Places

1 INTRODUCTION

• The paper uses “disability” in the ICF sense: an interaction between an individual’s impairment and restricted access to society.The introduction motivates MAP by noting that the safety and usefulness of multimodal AI assistance for real-world planning remain unclear.Source: §1 · p. 1
• MAP evaluates multimodal AI assistants for accessibility-aware visits to real-world places, covering mobility, vision, hearing, and cognition requirements.Its assessments compare system responses with ground-truth accessibility-feature data, using automated evaluation and human checks.Source: §1 · p. 1
• MAP introduces claim verification for accessibility-related place claims and visual evidence retrieval for images showing the requested place and feature area.The tasks assess response correctness and whether a model can find relevant visual evidence supporting an accessibility query.Source: §1 · p. 1, Figure 1 · p. 2
• Its time-based ground truth integrates public place and accessibility sources, with agent construction, human review, provenance tracking, and conflict resolution.Refreshed ground truth and fixed scoring procedures support system comparisons as places and accessibility information change over time.Source: §1 · pp. 1–2

2.1 Benchmark Design

• Unlike prior spatial and navigation benchmarks with preselected places and fixed ground truth, MAP evaluates up-to-date, factually correct accessibility information in an urban centre.Prior multimodal benchmarks include MMMU (Yue et al., 2024), MMBench, LiveBench (White et al., 2025), and MMIE (Xia et al., 2025); spatial and navigation examples include CitySeeker (Wang et al., 2025), V-IRL (Yang et al., 2024), and NavBench (Qiao et al., 2026).Source: §2.1 · p. 2
• MAP grounds its benchmark in real-world online accessibility information, collecting both textual and visual evidence for ground truth.This follows approaches such as LiveBench (White et al., 2025), which uses recently available information, and CitySeeker (Wang et al., 2025) and V-IRL (Yang et al., 2024), which use geospatial data and imagery.Source: §2.1 · p. 2

2.2 Assessing AI Systems on Accessibility

• Prior accessibility studies focus on particular disability types and application domains, while MAP evaluates multiple disability types using updated real-world location information across diverse features.Examples include auditing built-environment mobility with Accessibility SCOUT (Huang et al., 2025), virtual navigation with StreetReaderAI, and micro-navigation queries for blind and low-vision users.Source: §2.2 · pp. 2–3

2.3 Evaluation Methods and Metrics

• Prior multimodal recommendation commonly uses top-N accuracy, verification uses accuracy and F1, and evaluations often combine automatic metrics with human ratings or LLM-as-a-judge.Examples include AgentRecBench (Shang et al., 2026), AgentSelect (Shi et al., 2026), FACTS (Jacovi et al., 2025), RW-post, MUSCICLAIMS (Lal et al., 2025), SceneScout (Jain et al., 2025), and GuideDog (Kim et al., 2025); LLM-as-a-judge has shown results comparable with human evaluation (Pu et al., 2025).Source: §2.3 · p. 3

3 THE MAP BENCHMARK

• MAP evaluates responses to structured prompts against ground-truth place and accessibility-feature information in two tasks: claim verification and visual evidence retrieval.Claim verification checks feature-check and search-by-feature responses; visual evidence retrieval tests selection of evidence for a requested place and accessibility feature.Source: §3 · p. 3

3.1 Structured Prompts

• Prompts combine disability and accessibility requirements with feature-check, search-by-feature, or visual-evidence questions across Zurich places.Visual retrieval presents 20 candidate images, including a ground-truth image and controlled distractors, and checks both the correct POI and visibility of the requested feature. The three prompt sets cover 59 accessibility features, 257 place types, and 34 local areas.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 Ground Truth Data

• Ground truth represents a finite set of POIs with place attributes and accessibility features, assembled from place-information and accessibility-specific sources and reviewed by humans.Sources include Google Places, Ginto, official websites, targeted search results, user reviews, and POI images; the workflow aims to retain current, relevant information.Source: §3.2 · p. 4

3.2.1 Places

• The first MAP iteration builds a Zurich POI reference list to cover relevant place types across target areas.Records are matched and merged into POI entries while retaining distinct visitor destinations within larger buildings.Source: §3.2.1 · p. 4

3.2.2 Accessibility Features

• A single accessibility schema covers mobility, vision, hearing, and cognition, integrating evidence into field values such as booleans, text, measurements, or nulls.An agent weighs relevance, recency, source type, and agreement when reconciling evidence.Source: §3.2.2 · pp. 4–5

3.2.3 GT Construction and Updating

• Human reviewers fill missing accessibility values by inspecting sources, and MAP refreshes ground-truth data shortly before scheduled evaluations.Provenance records retain source details, timestamps, reconciliation notes, and conflicting information for auditing field selections and investigating model errors.Source: §3.2.3 · p. 5

4.1 Claim Verification for Accessibility Planning

• Claim verification pairs feature-check prompts for a named place with search-by-feature prompts to find qualifying places, assessing whether POIs and accessibility claims match ground-truth evidence.Full runs use automatic rating, with human assessment checking a proportion of responses.Source: §4.1 · p. 5, §4 · p. 5

4.1.1 Auto-rating

• Unverified was the dominant label; search increased Safe counts for several closed-weight systems but also increased Contradictory labels for most, while open-weight systems remained mostly Unverified.Retrieved evidence more often elicited cautious rather than Safe responses from open-weight systems.Source: §4.1.1 · p. 7
• With search, GPT-5.2’s Useful and Safe responses rose from 17 to 141, while search also increased Useful but Potentially Unsafe outputs.Gemini 3.1 Pro Preview had over 200 Useful and Safe responses with search; despite these gains, cautious and Not Useful responses remained prevalent.Source: Table 2 · p. 6, §4.1.1 · p. 7

4.1.2 Human Assessment

• Two humans’ assessments aligned with auto-rating for GPT-5.2, Claude Sonnet 4.6, Claude Sonnet 5, Gemini 3.6 Flash, and open-weights models; widest label-allocation gaps involved Grok 4.3 and Grok 4.5.The claim-verification scoring comparison used n=30. Scoring challenges included unprompted features, contradictions, and insufficient specificity.Source: §4.1.2 · pp. 7–8

4.2 Visual Evidence Retrieval for Accessibility Planning

• Visual retrieval asks systems to select an image or None for a specified place, feature, and visitor need, scoring whether the image shows the correct place and relevant physical area.A 0–1 score reflects place correctness and feature-area visibility; high scores indicate the correct place with the relevant area in focus.Source: §4.2 · p. 8
• More than two thirds of cases yielded Correct Visual Evidence for Gemini 3.6, Gemini 3.1 Pro Preview, GPT-5.6 Terra, and Claude Sonnet 4.6.Across 300 test prompts without search, responses also showed two common error patterns: selecting None or choosing images of the right feature area at the wrong place; results varied by feature.Source: §4.2 · p. 8, Table 4 · p. 8

5 LIMITATIONS

• Digital ground truth overrepresents visible physical-access features, while sources may lag behind changes to venues and visitor support.MAP refreshes ground truth before scheduled evaluation runs, but remains dependent on source timeliness and coverage.Source: §5 · p. 8
• MAP prompts use American English and may retain synthetic wording unlike the shorter, underspecified, or error-prone requests people use in practice.The prompt sets may not reflect regional, cultural, or multilingual language use.Source: §5 · p. 9
• MAP covers only features specified in its schema and assessable for scoring, excluding possible out-of-scope, temporary, or journey-level access requirements.Journey-level requirements concern the route from a source to a destination.Source: §5 · p. 9
• The first MAP iteration and its ground truth are limited to Zurich, so results are evidence for that city rather than a direct measure across regions.Source: §5 · p. 9

6 CONCLUSION

• MAP assesses multimodal AI systems as assistants for users with accessibility requirements when planning visits to real places.Source: §6 · p. 9
• Its two novel tasks assess factual accuracy in system responses and whether presented visual evidence is relevant to the user’s request.Source: §6 · p. 9
• The benchmark supports open-world evaluation by updating ground-truth data and running assessments within specified time windows.Source: §6 · p. 9

1 QUESTION TYPES AND SYSTEM PROMPTS

• At inference, task prompts are served as user requests to align evaluation with how human users query systems.Source: §1 · p. 11
• The appendix distinguishes task prompts, presented as user requests, from system prompts, supplied separately to guide responses.Source: §1 · p. 11
• A separate system prompt instructs models to produce outputs suitable for evaluation.Source: §1 · p. 11
• System prompts are concise and standardized across the evaluated models.Source: §1 · p. 11
• Table 5 presents sample prompts for each question type in claim verification and visual evidence retrieval.The table lists elements that make up the task prompts.Source: Table 5 · p. 11, §1 · p. 11
• Table 6 lists the system prompts used for evaluation runs by question type.Source: Table 6 · p. 11

• Validation-partition results for the three question types across claim verification and visual evidence retrieval appear in Tables 7–9.Source: §2 · p. 11

Human Assessments

• A response that correctly addresses the requested feature can become “Contradictory” when it adds a conflicting claim about a wheelchair-accessible entrance.This case illustrates how an additional claim can change the overall score despite a correct answer to the requested feature.Source: §3 · p. 11
• For wheelchair-accessible seating, the broad phrase “wheelchair accessible” is ambiguous because it does not specify seating rather than another feature.The assessed subset contained this response type only for wheelchair-accessible seating.Source: §3 · p. 11
• Auto-rater validation results cover 300 feature-check questions and 200 search-by-feature prompts in the claim-verification task.The search-prompt table reports counts for labels in the Verified/supported framework, including Useful and Safe, Safe but Uninformative, and Not useful.Source: Table 7 · p. 12, Table 8 · p. 12
• A response with correct restroom and seating claims can be labeled “Unverified” if it also claims parking that is absent from the ground-truth data.When “Safe”, “Unverified”, and “Contradictory” labels all apply, the final label is “Contradictory”, reflecting the potential consequences of incorrect information.Source: §3 · p. 12

4 SPECIFICATIONS FOR EVALUATION RUNS

• Reported results use a single run, with prompts processed independently and without prior-prompt context.Claim verification is auto-rated against ground-truth values and provenance context.Source: §4 · p. 12
• Supported inference settings use temperature 0 and low reasoning; open-weight models run through Ollama on a Google Cloud VM with one 24 GB NVIDIA L4 GPU.Tools are disabled during inference, Qwen2.5-VL-32B-Instruct has a 500-token generation limit, and search conditions use current data from the Brave Search API.Source: §4 · pp. 12–13
• Claim verification uses Gemini 3.1 Pro Preview to map response claims to MAP fields and compare them with ground truth and evidence; visual retrieval scoring is deterministic.Claim assessment considers the full response, including unrequested accessibility features, while visual prompts contain 20 candidate images with a ground-truth sample and controlled distractors; Table 9 validation uses 100 prompts without search.Source: §4 · pp. 12–13, Table 9 · p. 13