Source-linked AI summary
ARCH-B: Architectural Representation, Comprehension and Hierarchy Benchmark
Kieran Sagar Parikh, Jose Luis Garcia del Castillo y Lopez
TL;DR
Whether multimodal models can recognize a building across photographs and architectural drawings remains poorly characterized. ARCH-B evaluates this ability with a curated, four-choice benchmark and a non-expert human baseline. Results show uneven model performance across archetypes and weak agreement between human and model difficulty rankings.
Problem
Existing multimodal benchmarks do not directly test whether models preserve building identity across photographs and architectural drawings.
Source: §1 · p. 1Method
ARCH-B provides 354 questions across 11 archetypes, generated from a building-linked architectural image corpus and selected through model-guided difficulty screening and manual validation.
Source: §1 · p. 1, §3.1 · p. 3Results
Models perform unevenly across archetypes, with mean accuracy highest on mixed-representation outlier detection (64.22%) and lowest on photograph-only outlier detection (41.22%) and floorplan-to-photograph matching (44.14%).
Source: §5.2 · p. 6Takeaways & Limitations
ARCH-B is a reusable stress test for visual correspondence and representation transfer across architectural media, not a general measure of architectural intelligence.
Source: §7 · p. 8The human study is a non-expert baseline: short sessions, payment unrelated to accuracy, and possible inexperience or unfamiliarity limit broader comparisons between people and models.
Source: §6.3 · p. 7
Abstract
from arXiv · showhide
Multimodal models increasingly interpret visual environments, but their ability to recognize the same building across photographs, floor plans, elevations, sections, and renderings remains poorly characterized. We introduce ARCH-B, a benchmark of 354 four-choice questions across 11 cross-representational archetypes, constructed from a building-linked corpus of 3.9 million architectural images using visually similar distractors, model-guided difficulty screening, and manual validation. We evaluate 25 multimodal models and collect 5,830 responses from non-expert human participants. Model accuracy ranges from 10.45% to 83.90%, compared with a human baseline of 35.35%. Models perform comparatively well on mixed-representation outlier detection and photograph matching, but remain weaker on floorplan-to-photograph correspondence. Human and model difficulty across archetypes is only weakly correlated (Spearman's (\rho=0.33)). Held-out evaluation confirms that the difficulty identified during screening generalizes beyond the curation models. ARCH-B provides a diagnostic evaluation of visual correspondence and representation transfer across architectural media.
1. Introduction
ARCH-B addresses the limited evaluation of whether multimodal models can preserve building identity across photographs and architectural drawings. It defines this cross-representational task, constructs a benchmark, and evaluates models alongside non-expert humans.
1. Introduction
- Existing multimodal benchmarks test reasoning and spatial understanding but do not directly assess building identity across photographs and architectural drawings.Architectural visual understanding also requires aligning partial views and abstract drawings with embodied views of space.Source: §1 · p. 1
- ARCH-B tests whether visual evidence identifies the same building across changes in viewpoint, abstraction, and representational convention.Its hierarchy spans local interior and exterior views to whole-building drawings such as elevations, sections, and floor plans.Source: §1 · p. 1
- ARCH-B contains 354 questions across 11 archetypes, isolating correspondence and identity across heterogeneous representations of the same building.The benchmark covers photographs, floor plans, elevations, sections, and renderings, rather than scene description alone.Source: §1 · p. 1
- The benchmark uses a large architectural image corpus, programmatic question generation, model-guided difficulty selection, manual validation, and balancing.Four strong multimodal models guide sample selection before manual review.Source: §1 · p. 1
- 25 multimodal models were evaluated; Gemini 3.1 Pro Preview scored 83.90%, while the non-expert human baseline was 35.35%.The paper reports substantial variation across archetypes and models.Source: §1 · p. 2
2. Related Work
ARCH-B differs from prior visual, spatial, and architectural benchmarks by testing whether building identity persists across representational conventions. Its construction pairs structured generation with similar distractors, model-guided difficulty filtering, manual validation, and held-out evaluation.
2.1. Multimodal visual and spatial reasoning benchmarks
- ARCH-B complements view- and time-based spatial benchmarks by testing whether an entire building remains identifiable across viewpoints and representational conventions.Matching plans, elevations, sections, and renderings requires transferring evidence between projective images and abstract drawings.Source: §2.1 · p. 2
2.2. Architectural imagery, drawings, and multimodal evaluation
- Architectural datasets and benchmarks address parsing, layout recovery, drawing literacy, spatial cognition, or professional knowledge, while ARCH-B tests identity across heterogeneous representations.Prior resources include floor-plan datasets and architecture or engineering benchmarks.Source: §2.2 · p. 2
2.3. Benchmark construction and difficulty control
- ARCH-B combines structured generation, visually similar distractors, model-guided difficulty filtering, manual validation, and held-out evaluation to address benchmark artifacts and test difficulty generalization.The construction targets building-level correspondence rather than generic benchmark shortcuts.Source: §2.3 · p. 2
3. Methods
ARCH-B is built from linked architectural imagery and uses archetype-specific generation, visual selection, model-guided screening, and manual review. Its release includes the finalized benchmark and evaluation tools.
3. Methods
- The methods combine an architectural image corpus, candidate-question generation, filtering with four multimodal models, and manual validation.This pipeline constructs the final benchmark from generated candidates.Source: §3 · p. 2
3.1. Corpus of Architectural Imagery
- The corpus contains 3.9 million images, including 58,000 floor plans, from 95,000 real-estate listings and 41,000 architectural projects.Images remain linked to their parent listing or project as a building record, enabling queries across images from the same source.Source: §3.1 · p. 3
3.2. Question Sample Generation
- Questions use a four-choice format with one correct answer, and archetype-specific rules determine image types and instructions.Generation queries building records containing the required image types, samples eligible reference and correct-answer images, and draws incorrect choices across the corpus.Source: §3.2 · p. 3
- To improve question difficulty, reference images are selected for visual diversity and incorrect choices from the three nearest OpenCLIP images outside the correct answer’s building record.Agglomerative clustering selects centroid-nearest representatives; similar distractors make answer choices more plausible.Source: §3.2 · pp. 3–4, Figure 3 · p. 4
3.3. LLM-in-the-loop Difficulty Selection
- Of 2,200 generated candidates, 1,013 questions answered correctly by at most half of four screening models entered manual review.The review pool contained 147 items at 0% model accuracy, 256 at 25%, and 610 at 50%.Source: §3.3 · p. 4
3.4. Manual Curation for Final Benchmark Composition
- Two researchers reviewed every released question for a single correct answer, appropriate image types, and solvability; disagreements were resolved by consensus.Duplicate images, mismatched image types, and items either reviewer judged unsolvable were grounds for rejection.Source: §3.4 · p. 4
3.5. Release Protocol
- The release includes finalized question samples, answer choices, archetype metadata, and source links, but does not redistribute third-party image files.A runner verifies image hashes and supports evaluation; the benchmark is versioned, with corrections documented in a changelog.Source: §3.5 · pp. 4–5
4. Results
Across 25 models, ARCH-B produces a wide accuracy range and uneven performance across architectural correspondence tasks. Question-level results show graded difficulty, and held-out evaluation finds that screening-model difficulty trends generalize to other models.
4.1. Benchmark Composition
- 354 questions across 11 archetypes form the finalized benchmark, generated and curated using the methods described in Sections 3.1–3.4.The composition is summarized by archetype, including final question count and dominant task.Source: §4.1 · p. 5
4.2. Benchmark Evaluation Protocol
- Each model received the same ordered choices and image inputs, and correctness required an exact match between the parsed answer ID and the ground-truth ID.Images were resized to a maximum dimension of 1,024 pixels and re-encoded as JPEG at 90% quality; explanations did not affect scoring.Source: §4.2 · p. 5
4.3. Main Leaderboard
- 10.45%–83.90%: accuracy across 25 models on the full 354-question benchmark, with Gemini 3.1 Pro Preview highest at 83.90%.Most models had valid-response rates above 94%, but substantially lower rates for several models partly explain their low aggregate accuracies.Source: §4.3 · p. 5, Table 1 · p. 5
4.5. Per-Question Difficulty Distribution
- 4.00%–96.00%: question-level accuracy across models, with no question answered correctly by every model or missed by every model.145 of 354 questions were answered correctly by at most half of the models, indicating a graded range rather than uniform ease or impossibility.Source: Figure 4 · p. 6, §4.5 · p. 6
4.4. Performance by Archetype
- Models performed strongest on mixed-representation outlier detection and weakest on floorplan-to-photograph matching and photograph-only outlier detection.Other strong categories included same-building photograph matching, view selection from mixed drawings, and elevation-to-photo matching.Source: §4.4 · p. 6
4.6. Held-Out Model Analysis
- Held-out accuracy rose from 28.65% for questions answered by 0/4 screening models to 41.38% for 1/4 and 59.02% for 2/4.The authors report that screening difficulty generalized to the broader model set rather than reflecting model-specific weaknesses.Source: §4.6 · p. 6
5. Human Study
The human study establishes a non-expert baseline from 5,830 responses to ARCH-B questions. Human and model performance differ across archetypes, with only weak agreement in their relative difficulty rankings.
5.1. Human Study Design
- The study recruited English-speaking U.S. adults through Prolific for short sessions of 10 questions from one archetype each.The study was conducted in June 2026, and participants were paid $2 per session.Source: §5.1 · p. 6
5.2. Results
- 5,830 valid responses yielded a response-weighted human accuracy of 35.35% and macroaverage question accuracy of 35.37%, both above the four-choice chance level of 25%.Participants completed 583 sessions, with 16–18 human responses per question.Source: §5.2 · p. 6, §5.1 · p. 6
- Spearman’s ρ = 0.33: human and mean-model accuracy had weak rank correlation across archetypes, indicating limited agreement about which tasks were easier or harder.Human accuracy was highest for elevation-to-photograph matching (56.15%) and lowest for photograph-only outlier detection (21.61%) and exterior-from-interior matching (29.70%).Source: §5.2 · pp. 6–7
- Best observed model accuracy exceeded human accuracy in every archetype by 40.38–68.00 percentage points.Gemini 3.1 Pro Preview produced the best result for four archetypes, Gemini 3.5 Flash for three, and GPT-4.1 for two.Source: §5.2 · p. 6
6. Discussion
ARCH-B’s value lies in comparing performance across models and architectural task families, not in treating one topline score as a general measure of architectural intelligence. The discussion connects uneven performance to task demands and describes boundaries of the non-expert human comparison.
6.1. Benchmark Saturation and Continued Use
- A high topline score does not remove the need to diagnose which representational transitions remain difficult or whether gains generalize across task families.The authors describe ARCH-B as a comparative benchmark whose breadth and diagnostic structure reveal differences across models and task types.Source: §6.1 · p. 7
- The generation pipeline could produce targeted training or fine-tuning examples for weak task families rather than treating architectural visual understanding as one undifferentiated capability.The authors give floorplan-to-photograph matching and elevation-to-plan correspondence as examples.Source: §6.1 · p. 7
6.2. Human and Model Failure Modes
- Models were more accurate selecting a floor plan from photographs (58.78%) than selecting a photograph from floor plans (44.14%), while human accuracy was similar in the two directions (34.74% and 36.41%).The authors suggest models may use candidate-plan topology more readily than project an abstract plan into visual appearance; the directional asymmetry was largely absent for humans.Source: §6.2 · p. 7
6.3. Human Study Improvements and Future Work
- The human results are a non-expert benchmark baseline, not a general comparison between people and models under all scenarios.Short sessions and payment for completion rather than accuracy mean low performance may also reflect inexperience, limited attention, or unfamiliarity with plans and elevations.Source: §6.3 · p. 7
- Future versions could add other architectural media, move beyond fixed answer choices, and control distractor similarity, drawing style, image quality, building type, and geographic source.Proposed additions include axonometric drawings, site plans, construction documents, and three-dimensional models.Source: §6.3 · p. 8
7. Conclusion
ARCH-B evaluates whether multimodal models can align multiple architectural representations as views of the same building. The paper presents it as a reusable stress test for visual correspondence and representation transfer, rather than a general measure of architectural intelligence.
7. Conclusion
- ARCH-B evaluates alignment of photographs, plans, elevations, sections, and mixed evidence as representations of the same building.Evaluation across 25 models and a non-expert human baseline showed substantial differences among systems and uneven performance across tasks and questions.Source: §7 · p. 8
A. Question Archetypes
The section defines eleven four-choice archetypes that test building correspondence within photographs and across photographs, floor plans, elevations, sections, and interiors. Tasks include matching, identifying a view, and detecting an outlier.
A. Question Archetypes
- Floorplan–photograph matching runs in both directions: choose a plan for photographed buildings or choose a photograph for buildings shown in plans.The archetypes contain 23 and 56 questions, respectively.Source: A. Question Archetypes · p. 8
- Photograph-only tasks ask participants to identify an outlier among four images or select a photograph matching three reference photographs.The archetypes include 46 outlier questions and 24 inlying-image questions.Source: A. Question Archetypes · p. 8
- Interior–exterior correspondence is tested in both directions, selecting an interior from exterior references or an exterior from interior references.The two archetypes contain 25 and 20 questions, respectively.Source: A. Question Archetypes · p. 8
- Floorplan–elevation correspondence is also tested bidirectionally, selecting an elevation from plans or plans from elevations.These archetypes contain 32 and 30 questions, respectively.Source: A. Question Archetypes · p. 8
- Mixed-representation outlier detection presents one floor plan, elevation, section, and photograph, and asks which does not depict the same building.This archetype contains 37 questions.Source: A. Question Archetypes · p. 8
- Two archetypes require choosing a photograph from elevation drawings or from a combination of floor plan, elevation, and section drawings.The elevation-only and mixed-drawing tasks contain 32 and 29 questions, respectively.Source: A. Question Archetypes · p. 8