PaperlayerView exact revision on arXiv

Source-linked AI summary

IFC-MCQ: Automatically Generating Multiple-Choice Benchmarks for Evaluating LLM Agents on IFC Data Retrieval

Rabindra Lamsal, Sisi Zlatanova, Xuesong Shen

arXiv:2610.30533v1cs.CL

TL;DR

Evaluating natural-language IFC retrieval is difficult because open-ended answers require costly human review or uncertain LLM grading. IFC-MCQ generates and verifies multiple-choice benchmarks from IFC files, and its evaluation finds that coverage and agent performance vary with source models and retrieval interfaces.

  • Problem

    Natural-language IFC retrieval lacks scalable evaluation because free-text answers require costly human review or uncertain LLM grading, and existing approaches do not provide deterministic answer-level grading for new files.

    Source: §1 · pp. 2–3, §2.3 · p. 8

  • Method

    IFC-MCQ generates IFC-file-based multiple-choice items across a 15-category, 36-subtype taxonomy, independently re-derives answers, and retains verified items for exact option-set grading.

    Source: §3.1 · p. 10, §3.2 · p. 13, §7 · p. 55

  • Results

    Benchmark coverage varied with source-model content, and agent accuracy varied across models and retrieval interfaces, with category-level weaknesses apparent despite high overall accuracy in Stage 1.

    Source: §7 · p. 56

  • Takeaways & Limitations

    Category-level patterns can identify retrieval weaknesses relevant to construction applications that high overall accuracy may obscure.

    Source: §7 · p. 57

    IFC-MCQ scores reflect its generated multiple-choice setting, not retrieval alone or open-ended QA, and near-ceiling tool-agent accuracy limits fine-grained comparison.

    Source: §6.4 · p. 54

Abstract

from arXiv · show

Industry Foundation Classes (IFC), the standard for building information exchange in architecture, engineering, and construction (AEC), is increasingly queried through large language models (LLMs) using natural language. Evaluating these systems at scale is difficult because free-text answers require costly human or LLM-based review. We propose IFC-MCQ, a method that automatically generates multiple-choice benchmarks from IFC files for exact option-set grading. The taxonomy covers 15 categories and 36 subtypes across four groups, with answer keys derived from the file and distractors constructed from file-derived values or predefined rules. We applied IFC-MCQ to IFC models of varying sizes and modelling characteristics to assess generation coverage and evaluated two agentic configurations on selected models using tool- and code-based retrieval. The evaluation exposed category-specific failure modes and showed that performance can vary with the source IFC model and retrieval interface. IFC-MCQ enables scalable benchmarking of LLM-based IFC retrieval in AEC workflows.

1. Introduction

Natural-language access to IFC models could broaden who can retrieve building information, but complex schemas and the cost of grading open-ended answers make reliable evaluation difficult. IFC-MCQ generates verified multiple-choice benchmarks from IFC files, then uses them to examine model coverage and agent performance.

1. Introduction

  • IFC querying requires custom toolkit scripting or detailed knowledge of a complex, deeply nested schema, despite its use by a growing range of AEC users.Examples include facility managers querying room counts and energy consultants checking wall thermal transmittance.Source: §1 · p. 2
  • Open-ended IFC retrieval makes evaluation costly because free-text answers require human review or a second LLM whose judgments add uncertainty and inference cost.A plausible but incorrect answer, such as a fire rating, unit, or storey assignment, can affect downstream decisions.Source: §1 · pp. 2–3
  • MCQ grading requires exact matching between selected options and the correct answer set, but the paper reports no established MCQ benchmark for natural-language IFC querying.The proposed benchmark addresses the need for an IFC question taxonomy, direct item generation, and separate correctness verification.Source: §1 · p. 3
  • IFC-MCQ defines 15 question categories and 36 subtypes across four groups, targeting questions with reliable IFC-file ground truth.The groups are Direct retrieval, Relational, Quantitative, and Geometric; the taxonomy is intended to be extensible.Source: §1 · p. 3
  • Category generators derive question stems, answer keys, and IFC-specific distractors from file content, while a separate path re-derives answers and discards disagreements.Distractors can reflect values from other entities, incorrect units, or plausible values for absent properties.Source: §1 · p. 4
  • The evaluation first tests 17 LLMs with a tool agent on two reference models, then examines five selected LLMs across six models using tool and code agents.The later stages investigate source-model characteristics and retrieval-interface effects.Source: §1 · p. 4

2. Literature Review

Prior IFC interfaces progress from semantic parsing to LLM-based graph, database, and code-execution approaches, while evaluations often rely on limited question sets or non-deterministic grading. Existing benchmarks leave a gap for generating verified questions from new IFC files and grading answers deterministically.

2.1. Natural language querying systems for IFC

  • Early IFC interfaces translate natural-language questions into formal queries, but the reviewed semantic-parsing approaches are limited when questions require more complex reasoning.Examples include keyword mapping to IFC entities, voice-based syntax and semantics analysis, and a two-stage Text-to-BIMQL approach.Source: §2.1 · p. 5
  • Recent IFC systems use graph-based retrieval and LLM-generated queries or code, while BIMCoder instead translates language into software query statements rather than retrieving user-facing answers.Reviewed systems include ASK-BIM, Neo4j/Cypher approaches, multi-agent retrieval, and adaptive Python-code exploration [10, 11, 12, 14, 16, 18].Source: §2.1 · pp. 5–6

2.2. Benchmarks for IFC querying

  • IFC benchmarks range from free-text QA sets across multiple models to ground-truth extraction scripts and multi-session tasks, with free-text answers often requiring LLM grading.IFC-Bench v2 contains 1,027 pairs across 37 models, IFC-QA has 99 questions with extraction scripts, and IFCMemoryBench has 143 human-validated tasks.Source: §2.2 · p. 7

2.3. Summary and gap

  • The gap is a method that generates questions for new IFC files, verifies answer keys, and grades responses deterministically at the answer level; IFC-MCQ combines these capabilities.Its category coverage depends on whether the source file contains the relevant facts and distinct answer alternatives.Source: §2.3 · p. 8

3. Method

IFC-MCQ builds file-grounded multiple-choice benchmarks using a taxonomy that distinguishes retrieval mechanisms and question subtypes. Its pipeline generates candidate items from an IFC model, independently re-derives answers, and retains only items that pass verification and structural checks.

3.1. Question taxonomy

  • The taxonomy defines 15 categories and 36 subtypes across Direct retrieval, Relational, Quantitative, and Geometric groups, organized by the IFC mechanisms questions must read.The grouping supports reporting performance by required reasoning type and is intended to be extensible.Source: §3.1 · p. 10
  • Geometric questions are deliberately limited to lightweight operations on axis-aligned bounding boxes, leaving solid-geometry operations for future work.The scope supports extent comparison, base elevation, and point-in-box queries without solid-level operations.Source: §3.1 · p. 11

3.2. IFC-MCQ Pipeline

  • The four-stage pipeline builds a model index, generates candidate items, separately verifies their answers, and structurally checks verified items before retaining them.Each retained item includes an identifier, taxonomy labels, question type and text, options, answer, and source file.Source: §3.2 · p. 13

3.2.1. Model Index and Category Generators

  • The model index defines current benchmark scope by collecting IFC facts and relationships that generators can query or derive from.The index scope can be extended by adding further lookups.Source: §3.2.1 · pp. 13, 15
  • Category generators form fact pools from the index, fill question templates, and create distractors that reflect plausible IFC retrieval errors.Distractors use file-derived alternatives where possible, with fixed transformations used when distinct options are otherwise unavailable; candidates are skipped if valid distinct options cannot be formed.Source: §3.2.1 · pp. 15–17

3.2.2. Verifiers and Structural Checks

  • Verifiers re-open the IFC file and independently re-derive answers from candidate metadata, while structural checks reject malformed items or distractors that match the answer.Generation and verification use the same IfcOpenShell build, so verification does not guard against a library bug.Source: §3.2.2 · pp. 17–18, §3.2.1 · p. 17

4. Implementation and Evaluation Setup

The evaluation applies IFC-MCQ to ten parseable IFC models and tests tool- and code-based agents under a shared scoring protocol. It combines a staged evaluation across models and retrieval interfaces with a stratified manual audit of generated items.

4.1. IFC Models

  • Generation uses ten IFC models, a random seed of 10, and a cap of 40 verified items per category per model; the cap balances category representation rather than limiting the method theoretically.A 10-item reporting threshold identifies limited category representation; single-choice questions have two to four distractors, and multi-choice questions at least one.Source: Figure 2 · p. 19, §4.1 · p. 20, Table 4 · p. 20

4.2. Agentic System Evaluation

  • Both agents inspect the model before answering and follow the same protocol, with up to eight interaction steps and an answer-only fallback if no parseable answer is produced.The setup holds prompts, protocol, and scoring constant across models.Source: §4.2 · p. 21
  • The tool agent uses 17 read-only tools, while the code agent writes and executes IfcOpenShell code to retrieve information directly from the model.Tool observations and code outputs are limited to 6,000 characters; the tool set provides at least one route for every category in the benchmark scope.Source: §4.2 · pp. 21, 23, Figure 3 · p. 22

4.2.1. Scoring and metrics

  • Answers count as correct only when selected options exactly match the answer key, and accuracy is reported overall and across taxonomy levels against two chance baselines.The baselines always select the first option or select options at random.Source: §4.2.1 · pp. 26–27

4.2.2. Evaluation Protocol

  • The three-stage protocol first evaluates 17 LLMs with the tool agent on two reference models, then evaluates five selected LLMs across six models with tool and code agents.Stages 2 and 3 use the same 890 items, question text, options, and option ordering; the LLMs and models are held fixed when comparing retrieval interfaces.Source: §4.2.2 · pp. 27–29

4.3. Human audit of generated items

  • A stratified audit sampled one item per represented model-category combination and checked question wording, independently derived answers, and unique answer-option representation.The audit used 89 items across six models.Source: §4.3 · pp. 29–30

5. Results

The results section reports benchmark generation and verification, then the agentic evaluation across models and retrieval interfaces, and finally a no-IFC-access control.

5.1. Benchmark Generation, Verification, and Human Audit

  • The results begin with generation and verification across ten IFC models, alongside a stratified human audit.Source: §5 · p. 30

5.2–5.3. Tool agent evaluation

  • The agentic evaluation first tests the tool agent on two reference models, then examines its performance across six models.Source: §5 · p. 30

5.4–5.5. Agent comparison and control

  • The section next compares tool and code agents using the same six models, LLMs, and benchmark items, before reporting a no-IFC-access control.Source: §5 · p. 30

5.1. Benchmark Generation, Verification, and Human Audit

IFC-MCQ generated and verified broad but model-dependent benchmark coverage, and all items in the stratified human-audit sample passed its checks. Verification disagreements across heterogeneous models exposed implementation and modelling edge cases.

FZK Haus and Duplex

  • 1,177 items were generated and all passed verification for FZK Haus and Duplex, representing 35 of the taxonomy’s 36 subtypes.The SPA storey’s building subtype had no items because each model contained only one building, while QTY and AGG were limited by admissible quantity facts.Source: §5.1 · pp. 30–31

Additional IFC models

  • Verification disagreements involved type-level properties, parent-associated assembly components, subclass aggregation, and differently resolved same-named property sets.These cases illustrate how heterogeneous models can expose edge cases not apparent in smaller reference models.Source: §5.1 · pp. 31, 34
  • 4,467 of 4,482 candidates from the eight additional models passed verification; coverage varied, with some categories sparse or absent when models lacked suitable facts or relationships.Fifteen candidates were rejected after separate answer re-derivation disagreed with the generator.Source: §5.1 · p. 31, Table 7 · p. 33

Human audit results

  • All 89 sampled items passed the manual checks for question wording, independently derived answers, and unique representation of the correct option set.No sampled item required correction or removal.Source: §5.1 · p. 34, Table 8 · p. 34

5.2. Stage 1: Tool agent on FZK Haus and Duplex

On the 1,177-item Stage 1 benchmark, every evaluated LLM outperformed chance, but accuracy varied by model and question difficulty. Direct lookups were generally easier than quantitative, geometric, and other tasks involving missing values or exact answer-set matching.

5.2.1. Overall Performance

  • 14 of 17 LLMs exceeded 95% overall accuracy, seven exceeded 99%, and two answered every item correctly; dummy and random baselines scored 28.0% and 27.5%.The two perfect-scoring models were qwen3.7-max and qwen3.6-27b.Source: §5.2.1 · pp. 34–35

5.2.2. Performance by group and question type

  • Direct retrieval exceeded 97% for nearly all models, while quantitative and geometric questions were harder, especially for gpt-5.4-nano and gemma-3-27b-it.Those models scored 86.2% and 84.8% on Quantitative, and 71.2% and 68.8% on Geometric, respectively.Source: §5.2.2 · pp. 35–36, Figure 6 · p. 36
  • Multi-choice items were less accurate than single-choice and boolean items because every correct option had to be selected without any distractor.Five models, including qwen3.7-max and gemma-4-31b-it, scored 100% on multi-choice items.Source: §5.2.2 · pp. 36–37

5.2.3. Category and Subtype-level Analysis

  • Figure 7 shows category averages led by Attribute retrieval and Class identification at 99.3%, while Existence / absence averaged 86.3% and Comparison / extremum 86.2%.Across the 16 models other than llama-3.1-8b-instruct, those two difficult categories also had the largest error counts: 122 and 109, respectively.Source: §5.2.3 · pp. 37–38, Figure 7 · p. 38
  • Existence / absence errors often substituted plausible sibling values for missing properties, while extremum errors involved memory, tool-use, and answering-protocol failures.Value-or-null items accounted for 94 of 122 existence errors; largest and smallest items accounted for 83 of 109 comparison errors.Source: §5.2.3 · pp. 38, 40
  • Space-boundary errors often involved over-selecting options, whereas geometry errors centered on comparing coordinates from two boxes rather than choosing tools.Extent compare and base elevation require extracting an axis extent or minimum Z value and comparing it; point-in-box was comparatively easy once the bounding-box tool was called.Source: Figure 8 · p. 39, §5.2.3 · pp. 41–42
  • Counting errors arose from miscounting host-specific openings or selecting the wrong type count, while spatial-structure and type-resolution lookups were mostly correct once the relevant tool was called.Direct lookups were stronger for some models than tasks requiring missing-information checks, multi-set matching, or reasoning over observations; llama-3.1-8b-instruct performed poorly across nearly every category.Source: §5.2.3 · pp. 42–43

5.3. Stage 2: Tool agent on additional IFC models

Stage 2 tested five selected LLMs with the tool agent on four additional IFC models as well as the two Stage 1 references. Accuracy was lower on the additional models, with errors reflecting category-specific challenges and model-dependent retrieval conditions, including truncated tool outputs.

5.3. Stage 2: Tool agent on additional IFC models

  • On the four additional IFC models, qwen3.6-27b scored 98.0%, gpt-5.6-luna 97.6%, and gpt-oss-120b 97.1%, while deepseek-v4-flash and gemma-3-27b-it scored 93.9% and 82.7%.The latter two fell from 99.7% and 87.0%, respectively, on the FZK Haus and Duplex items; Table 9 reports tool-agent overall performance.Source: §5.3 · p. 43, Table 9 · p. 44
  • The five models had no errors in Class identification, Attribute retrieval, or Spatial structure, while errors clustered in Aggregation, Counting, Existence / absence, Comparison / extremum, and Geometry.Trapelo and TNW accounted for 117 of the 181 tool-agent errors.Source: §5.3 · p. 44
  • Lower accuracy on the additional models reflected category-specific difficulties and model-dependent retrieval conditions, not model size alone.A 6,000-character tool-output limit truncated results for an average-height query over 522 walls; deepseek-v4-flash also repeated an absent-property hallucination and similar geometry errors from Stage 1.Source: §5.3 · pp. 44–45

5.4. Stage 3: Code agent versus tool agent

Replacing the tool agent with code execution changed accuracy differently across LLMs and models; the category-level comparison also exposed areas where performance diverged. Class identification and attribute retrieval remained strong, while geometry showed the clearest difference after excluding one LLM.

5.4. Stage 3: Code agent versus tool agent

  • Accuracy changes with the code agent varied by LLM and source model, including lower scores for three LLMs on both model groups and mixed results for two others.gpt-5.6-luna answered five fewer items correctly on FZK Haus and Duplex but matched accuracy on the four additional models; deepseek-v4-flash fell on the reference models and rose on the additional models.Source: §5.4 · pp. 45, 47
  • Across all five LLMs, the code agent had its largest reductions in openings and fillings, geometry, spatial structure, material association, and space boundaries.Moderate reductions occurred in type resolution, quantity retrieval, and boolean property; property retrieval, existence/absence, counting, and aggregation improved, while comparison/extremum was unchanged.Source: §5.4 · pp. 45, 47, Table 12 · p. 46
  • 80.8% versus 95.8% accuracy for the code and tool agents, respectively, remained the largest category difference after excluding gemma-3-27b-it.Class identification and attribute retrieval stayed at or near 100% with both agents; differences in several other categories narrowed after the exclusion.Source: §5.4 · pp. 45, 47, Table 12 · p. 46

5.5. No-IFC-access control

Without IFC access, the four LLMs scored 37.0–49.7%; the three also tested with agents scored higher when given model access. Because the control does not isolate IFC access from agent interfaces and interaction, its results do not measure retrieval alone.

5.5. No-IFC-access control

  • 37.0–49.7% accuracy across four LLMs shows that questions and options alone yielded substantial correct answers without access to the source IFC models.gpt-5.6-luna and qwen3.6-27b each reached 49.7%, gemma-3-27b-it 38.3%, and llama-3.1-8b-instruct 37.0%.Source: §5.5 · pp. 47–48, Table 13 · p. 48
  • Boolean items scored higher than multi-choice items for all four models, but their two options also set a 50% random-selection baseline.The authors caution that the access comparison is not a pure IFC-access ablation because agent configurations add a retrieval interface and iterative interaction.Source: §5.5 · p. 48
  • From 49.7% to 98.4% with the tool agent and 97.9% with the code agent, gpt-5.6-luna improved; qwen3.6-27b rose from 49.7% to 98.7% and 95.7%, and gemma-3-27b-it from 38.3% to 84.2% and 65.1%.The first value in each pair is the no-access control; the next values are tool-agent and code-agent accuracy.Source: §5.5 · p. 48, Table 13 · p. 48

6. Discussion

The discussion interprets agent performance alongside interaction costs, benchmark coverage, and methodological limits. It emphasizes that results depend on the source IFC model and retrieval interface, and that the benchmark’s scope and grading setting constrain interpretation.

6.1. Agent interaction and resource usage

  • For tool-agent systems above 95% accuracy in Stage 1, 82–88% of items were answered in one call, with 1.15–1.33 mean calls per item; llama-3.1-8b-instruct averaged 6.01 calls and scored 30.9%.In Stages 2 and 3, the tool agent answered 78.0–87.9% of items in one call across models, versus 60.9–66.6% for the code agent; observations were truncated at 6,000 characters for 186 tool-agent and 81 code-agent answers.Source: §6.1 · p. 49, Table 14 · p. 50
  • The code agent used more input tokens than the tool agent for every LLM except gpt-5.6-luna and more output tokens for every LLM.deepseek-v4-flash averaged 9,480 input and 3,276 output tokens per item with the code agent.Source: §6.1 · pp. 49–50, Table 15 · p. 50

6.2. Role of the retrieval interface

  • Across six models, 78.0–87.9% of tool-agent answers were completed after one call, consistent with tools designed to provide a retrieval route for every benchmark category.The tool set makes many retrieval tasks relatively direct, so near-ceiling accuracy should be interpreted in the context of that predefined interface.Source: §6.2 · pp. 50–51
  • Code execution changed accuracy in both directions across LLMs and categories, and generally required more steps and tokens; deepseek-v4-flash improved under code execution.The tool agent generally performed better on tasks with an intended route in its predefined tools, but code-agent performance beyond the current taxonomy was not evaluated.Source: §6.2 · p. 51, §6.3 · p. 53

6.3. Implications for benchmark generation and agentic evaluation

  • All retained benchmark items passed separate answer re-derivation, and a stratified human audit found well-formed sampled items with matching answers uniquely represented among the options.Some candidates from TNW, Trapelo, and Sample House were rejected after generated and re-derived answers disagreed.Source: §6.3 · pp. 51–52
  • Coverage depends on source-model content: missing relationships or too few distinct values can prevent categories from yielding items or cause candidate-generation skips.TNW and UNSW Block House yielded no space-boundary items because they lacked IfcRelSpaceBoundary relationships; all 402 skips were in property retrieval or existence/absence.Source: §6.3 · p. 52
  • Category-level reporting can reveal weaknesses obscured by overall accuracy, while interface comparisons should use the retrieval configuration intended for deployment.Class identification and attribute retrieval were consistently high-accuracy, whereas other task types showed greater error concentrations.Source: §6.3 · p. 53

6.4. Limitations

  • Multiple-choice scores may depend on options and distractors, and template questions omit the full linguistic variation, ambiguity, and conversational context of real queries.The no-access control reached 37.0–49.7%; scores should not be treated as direct measures of retrieval alone or open-ended question answering, and near-ceiling tool-agent results limit fine-grained comparisons.Source: §6.4 · pp. 53–54

7. Conclusion

IFC-MCQ generates multiple-choice benchmarks from IFC files and verifies answers through a separate re-derivation path for exact option-set grading. Tests across models and agent configurations found that coverage and observed performance vary with source-model content and retrieval interface, with interpretation bounded by the benchmark’s scope.

7. Conclusion

  • IFC-MCQ defines 15 categories and 36 subtypes across four groups, generating stems, answer keys, and distractors from each input IFC file.A separate verification path re-derives answers without the stored key, enabling exact option-set scoring without human or LLM judges.Source: §7 · p. 55
  • Across ten IFC models, benchmark coverage varied with available relationships, quantities, and distinct values; most candidates passed verification, with a small number rejected after answer disagreement.A stratified audit found sampled items well formed, with independently retrieved answers matching keys and uniquely represented among options.Source: §7 · p. 56
  • Across the three-stage evaluation, accuracy varied across source models, LLMs, and retrieval interfaces, and the code agent generally required more interaction steps and tokens.Stage 1 tested 17 LLMs on two reference models; later stages tested five LLMs across six models with tool and code agents.Source: §7 · p. 56
  • Category-level analysis identifies retrieval types where systems performed reliably or made errors that overall accuracy can obscure.The paper highlights errors involving absent properties, incomplete relationship retrieval, counting or aggregation, and geometric comparison.Source: §7 · pp. 56–57
  • The reported findings are specific to multiple-choice evaluation, AABB-based geometry, the shared IFC-processing library, and the evaluated models and interfaces.Future work is proposed on broader geometry and multi-step retrieval, additional models and agent architectures, and transfer to free-text IFC question answering.Source: §7 · p. 57

Related papers

Loading 2610.30533v1…