PaperlayerView exact revision on arXiv

Source-linked AI summary

Build2SPARQL: A Large-Scale Text-to-SPARQL Benchmark Dataset for Building Knowledge Graph Querying

Wooyoung Jung

arXiv:2610.00224v1cs.AI

TL;DR

Building text-to-SPARQL research lacks a large benchmark despite operators’ need to query heterogeneous, complex building graphs in natural language. Build2SPARQL constructs a KG-grounded dataset with validated queries and register-varied questions; retrieved few-shot examples substantially improve exact-match performance across evaluated open-weight models.

  • Problem

    Building operators need natural-language access to heterogeneous, schema-complex building graphs, but large building-domain NL/SPARQL benchmarks are lacking.

    Source: Introduction · p. 2

  • Method

    Build2SPARQL mines and validates SPARQL queries from building KGs, then uses LLMs to generate five register-varied questions per query.

    Source: Introduction · p. 2, Conclusion · p. 21

  • Results

    Retrieved three-shot evaluation achieved Exact Match F1 of 56–65% across three open-weight LLMs, substantially above zero-shot baselines.

    Source: Conclusion · p. 22

  • Takeaways & Limitations

    The corpus supports few-shot natural-language-to-SPARQL research on building KGs.

    Source: Conclusion · p. 22

    Linear chains make up 81.6% of the corpus, which can skew models trained on its raw distribution toward simple retrieval.

    Source: Limitations · p. 20

Abstract

from arXiv · show

Building automation systems are increasingly represented as semantic knowledge graphs (KGs) using ontologies such as Brick and ASHRAE 223P, creating a machine-readable substrate for artificial-intelligence applications. One promising application is translating natural-language questions into SPARQL (text-to-SPARQL), which would let building operators query these graphs through language agents, but progress is limited by the scarcity of large natural-language/SPARQL benchmarks. This paper presents Build2SPARQL, a large-scale benchmark for building KGs generated by a KG-grounded pipeline: SPARQL queries are produced and validated entirely by graph-traversal code, while large language models generate only the natural-language questions, keeping query correctness independent of model behavior. The pipeline mines six query-pattern families -- linear chains, branching, UNION, aggregation, OPTIONAL, and attribute-filtered -- and phrases each query across five vocabulary registers. Applied to 201 building KGs (180 Brick, 21 ASHRAE 223P), it yields 6,136 executable SPARQL queries and 30,680 questions. A two-rater human validation of 300 questions found 98.8% semantic fidelity, 98.8% naturalness, and 84.0% operational plausibility. A retrieval-augmented evaluation across three open-weight language models raised exact-match accuracy from 0.2-20% (zero-shot) to 56-65% (three-shot retrieved).

Introduction

Building operators need natural-language access to building knowledge graphs, but building-domain text-to-SPARQL research lacks a large benchmark spanning diverse buildings and query patterns. Build2SPARQL addresses this gap with KG-grounded query generation and varied questions, complementing existing general- and building-domain resources.

Introduction

  • The benchmark gap matters because building graphs are heterogeneous and schema-complex, while operators need to express information needs without authoring SPARQL.Existing building-domain coverage was limited to 27 queries and 188 questions across four buildings.Source: Introduction · p. 2
  • Build2SPARQL pairs 6,136 executable queries with 30,680 questions across 201 building KGs, using six query families and five vocabulary registers.Graph-traversal code generates and validates queries; LLMs generate questions only after query verification.Source: Introduction · pp. 2–3

Benchmarks for Natural-Language/SPARQL: General Domain

  • General-domain benchmarks span thousands of pairs, but their encyclopedic schemas differ from building graphs’ equipment-centered structure and do not transfer without domain-specific data.The cited landscape includes LC-QuAD, QALD, Spider, and BIRD.Source: Benchmarks for… · p. 3

Building-Domain Benchmarks and QA Data Construction

  • BuildingQA combines expert-provided queries with LLM phrasings, offering ecological validity but only 27 queries across four buildings.Its questions have low direct term overlap with KG terms, reflecting functional descriptions of entities.Source: Building-Domain Benchmarks and QA Data… · p. 3
  • Other building benchmarks either remain small or use narrow template inventories, leaving broader pattern coverage and multiple registers as Build2SPARQL’s target.BuildingGPT2’s fixed templates omit branching joins, UNION, OPTIONAL, grouped aggregation, attribute filters, and multi-namespace traversal.Source: Building-Domain Benchmarks and QA Data… · pp. 3–4

Text-to-SPARQL Methods and Automated Generation

Prior text-to-SPARQL approaches trade structural coverage against executability or linguistic flexibility. Build2SPARQL instead discovers patterns from each graph, validates queries before LLM use, and generates questions across vocabulary registers.

Text-to-SPARQL Methods and Automated Generation

  • Template systems preserve correctness but restrict query structure, neural models risk invalid queries, and LLM prompting still faces limited building-domain training data.The related work reports a maximum F1 of approximately 38% on BuildingQA under few-shot prompting.Source: Text-to-SPARQL Methods and Automated… · p. 4
  • KG-grounded generation discovers patterns in the target graph and uses LLMs for register-specific questions only after query validation, avoiding fixed templates and query-structure errors.The stated corpus focuses on Brick-schema buildings; extension to other ontologies is future work.Source: Text-to-SPARQL Methods and Automated… · p. 5

Methodology

  • The five-stage pipeline mines patterns, validates executable queries, filters for relevance, refines queries, and generates natural-language questions.The ordering separates structural correctness from linguistic generation.Source: Methodology · p. 5

SPARQL Pattern Mining by KG-Grounded Breadth-First Search (BFS)

  • Pattern mining discovers available query structures at runtime from the target graph, with linear and branching patterns common and other types dependent on graph-specific co-occurrences.The six operators are linear chains, branching, UNION, COUNT/GROUP BY, OPTIONAL, and attribute filtering.Source: SPARQL Pattern Mining by KG-Grounded… · p. 6
  • Linear chains enumerate acyclic paths by breadth-first search, while branching paths join two distinct relationships from a shared anchor.A per-depth cap preserves multi-hop paths when dense graphs would otherwise exhaust the pattern budget with shallow paths.Source: SPARQL Pattern Mining by KG-Grounded… · p. 6
  • UNION pairs alternative routes to a shared terminal, COUNT groups linear paths, and OPTIONAL captures properties present on only some terminal instances.These pattern families require corresponding structural co-occurrences in the graph.Source: SPARQL Pattern Mining by KG-Grounded… · p. 6
  • Attribute-filtered paths create separate variants for measurement attributes, which in this corpus occur only in ASHRAE 223P buildings.The pattern uses quantifiable property annotations such as quantity-kind or unit values.Source: SPARQL Pattern Mining by KG-Grounded… · p. 6

Executable Validation by Triple Walking

Candidate queries are checked against the source graph, then filtered for operational relevance with priority given to patterns that expand coverage across underrepresented categories.

Executable Validation by Triple Walking

  • Python triple-walking traverses each candidate’s property path and discards patterns with no matching rows; matched tuples also ground later instance-specific questions.Equipment identifiers come from the actual building’s matching instances.Source: Executable Validation by Triple Walking · p. 7

Semantic Relevance Filtering with Coverage Prioritization

  • A binary Gemma 4 4B filter removes patterns judged trivial, too generic, or not actionable for building operation.The prompt asks for a YES or NO judgment with a short explanation.Source: Semantic Relevance Filtering with… · p. 7
  • Before filtering, candidates are ranked by underrepresented system type, ontology class, complexity, and pattern type so coverage-expanding patterns are evaluated first.The rarest-cell-first ordering is intended to protect underrepresented categories when budget limits apply.Source: Semantic Relevance Filtering with… · p. 7

Query Refinement and Summarization

The pipeline preserves query structure during refinement, then generates questions at five vocabulary registers using stage-specific local Gemma models and prompts designed to vary phrasing.

Query Refinement and Summarization

  • Gemma 4 4B rewrites variable names and summarizes retrieved information without changing query structure, followed by an execution check.Triple patterns, grouping clauses, OPTIONAL blocks, and UNION blocks are preserved.Source: Query Refinement and Summarization · p. 7

Multi-Register Natural-Language Question Generation

  • Five questions per validated query span ontology-literal technical wording through plain operational English rather than merely paraphrasing one utterance.The register prompts target distinct vocabulary levels that are not intended as categorically separable question types.Source: Multi-Register Natural-Language… · p. 7
  • Questions are conditioned on the building description, query, retrieved summary, and validation-walk results; Q5 withholds named equipment identifiers to avoid tag leakage.Previously generated questions are supplied with instructions to vary sentence structure and opening words.Source: Multi-Register Natural-Language… · pp. 7–8

Generation Configuration and Reproducibility

  • All generation stages use locally served open-weight Gemma 4 models without fine-tuning, with 4B models for filtering and query rewriting and 31B for question generation.The generation prompt also steers non-linear query phrasing, and Q4 receives named-instance examples from validation results.Source: Generation Configuration and… · p. 8

Build2SPARQL Dataset

Build2SPARQL provides executable queries and five natural-language questions per query across 201 building knowledge graphs. Its corpus combines real and synthetic buildings, with graph sizes and query counts reflecting substantial variation.

Build2SPARQL Dataset

  • Each query has five register-differentiated questions, and graph-traversal validation guarantees that the SPARQL is executable against its source graph.The corpus covers six query-pattern families across 201 buildings and 6,136 queries.Source: Build2SPARQL Dataset · p. 8

Corpus Overview

  • The 201 graphs comprise 180 Brick and 21 ASHRAE 223P models, including 81 real and 120 synthetic buildings.The real buildings include 47 Mortar Brick buildings, 13 curated Brick models, and 21 ASHRAE 223P examples.Source: Corpus Overview · p. 9
  • The 120 synthetic Brick graphs span eight DOE commercial archetypes and three construction vintages, with five instances in each type-vintage cell.They complement the organic heterogeneity of the real buildings with labeled structural diversity.Source: Corpus Overview · p. 9
  • Graphs range from 4 to 932 equipment instances; two unusually sparse Mortar buildings remain in the corpus with fewer than five valid queries each.Processing is capped at 45 minutes per building; the sparse counts are attributed to graph sparsity, not a pipeline limitation.Source: Corpus Overview · p. 9, Table 4 · p. 9
  • BuildGraph buildings cluster at 408–460 queries per type, while real buildings vary more widely in query counts.The reported difference reflects regular prototype structure in the synthetic graphs and heterogeneity in real-graph density and completeness.Source: Corpus Overview · p. 9
  • All queries use SELECT DISTINCT, and 98% use subclass-aware rdf:type/rdfs:subClassOf* paths to retrieve instances typed under an anchor class.The property path addresses Brick’s use of specific leaf classes; without it, queries can return semantically incomplete results.Source: Corpus Overview · pp. 9–10
  • Each record includes a building-context description and a plain-language retrieval summary, alongside exactly five questions.The retrieval summary supports retrieval-augmented prompting use cases.Source: Corpus Overview · p. 10

Pattern-Type Distribution

Linear chains make up most mined queries, while the questions span five registers from ontology-literal wording to plain language. The section also describes register-related question characteristics and the corpus’s validation approach.

Pattern-Type Distribution

  • The 81.6% linear share can dominate raw-distribution evaluation, so the authors recommend per-pattern reporting and targeted or balanced challenge sets.The passage also recommends oversampling or loss-reweighting rarer types for training.Source: Pattern-Type Distribution · p. 10

Natural-Language Question Characteristics

  • Q1 averages 13.8 words, Q2 16.2, Q3 14.2, Q4 15.5, and Q5 14.6, with Q2 the longest register.The passage attributes Q2’s length to multi-word engineering descriptors and Q3’s shorter phrasing to removal of schema vocabulary.Source: Natural-Language Question… · pp. 10–11

Pattern-Type Distribution

  • 81.6% of queries are linear chains, reflecting regular, repeating equipment hierarchies in the DOE prototype archetypes.Each of the five non-linear types requires structural co-occurrences met less frequently in the corpus.Source: Pattern-Type Distribution · p. 10, Table 5 · p. 11

Natural-Language Question Characteristics

  • 79% of Q4 questions are detected as instance-grounded, a conservative lower bound; Q5 withholds instance identifiers by design.The remaining Q4 cases include count-only or vocabulary-URI results and names that pattern matching cannot distinguish from generic terms.Source: Natural-Language Question… · p. 11
  • Q1–Q3 form a vocabulary-abstraction zone, while Q4–Q5 shift to whether the question names a specific equipment instance.Q1 uses ontology terms, Q2 engineering terminology, and Q3 operational language; Q4 names an instance, whereas Q5 omits it.Source: Natural-Language Question… · p. 11
  • The corpus uses 232 distinct Brick class and property terms across five subsystems, with its five most-queried terms reflecting an air-side HVAC focus.The top terms are hasPoint, VAV, feeds, HVAC Zone, and Air Handler Unit.Source: Natural-Language Question… · p. 11

Validation

  • Validation combines construction-time SPARQL executability checks, two-rater human review, and baseline NL-to-SPARQL evaluation across three open-weight LLMs.The corpus is also compared with BuildingQA across scale, pattern coverage, question quality, and pattern-type overlap.Source: Validation · p. 11

Executability Verification

The corpus’s queries are executable against their source graphs, and human review supports the quality of its questions. In the held-out evaluation, retrieved examples substantially improve exact-match results, with important register and building-type qualifications.

Executability Verification

  • Every corpus query is executable by construction: candidates with no result rows are discarded before question generation, and all ground-truth queries on 28 test buildings executed successfully.The source graphs and query-to-building metadata allow independent verification.Source: Executability Verification · p. 12

Per-Pattern-Type Quality Breakdown

  • Branching and OPTIONAL queries have the highest distinct-opening rates, at 0.910 and 0.883; Q4 instance-grounding rates are highest for linear and branching queries, at 81% and 82%.Aggregation and UNION have lower Q4 grounding rates (52% and 22%), and OPTIONAL is lowest (17%).Source: Per-Pattern-Type Quality Breakdown · p. 12

Human Validation of Natural-Language Questions

  • Human raters found 98.8% semantic fidelity, 98.8% naturalness, and 84.0% operational plausibility in the 300-question sample.Eleven of 12 plausibility failures involved sampled ASHRAE 223P connection-topology queries; every other sampled query passed in all five registers.Source: Human Validation of Natural-Language… · p. 13
  • Connection-topology queries make up 14.3% of the corpus and are valid, executable queries, but are less representative of routine operator questions; UNION results should not be read as operator-facing performance.The passage estimates corpus-level plausibility at about 87% and UNION-family plausibility at about 34%.Source: Human Validation of Natural-Language… · p. 13

NL-to-SPARQL Baseline Evaluation

  • 3-shot retrieved raises Exact Match F1 from under 21% to 56–65% across all three models, versus 0.2–20% for zero-shot.Random 3-shot reaches 22–35% Exact Match F1 when retrieved examples are unavailable.Source: Table 8 · p. 15, NL-to-SPARQL Baseline Evaluation · p. 15
  • Under retrieved 3-shot, Q4 scores 17–37% Exact Match F1 across models, below Q5 at 57–68%, because instance-named questions mismatch generic ground-truth queries.The passage attributes the gap to disjoint result sets when models produce instance-specific queries for Q4.Source: NL-to-SPARQL Baseline Evaluation · p. 15, Figure 2 · p. 16, Table 9 · p. 16
  • Retrieved 3-shot reaches 64–68% Exact Match F1 on linear chains, while UNION is hardest at 9–18%; non-linear pattern estimates are directional because their samples are small.The evaluation sample contains 513 of 555 rows for linear chains, versus 5–15 rows for each non-linear type.Source: NL-to-SPARQL Baseline Evaluation · p. 15, Table 10 · p. 16
  • Held-out BuildGraph hospitals score 67–74% Exact Match F1 versus 43–55% on real buildings, while a matched-modality oracle provides no advantage over realistic question-keyed retrieval.The authors associate the hospital result with shared procedural structure and identify real-building scores as the more meaningful indicator of deployment difficulty.Source: NL-to-SPARQL Baseline Evaluation · p. 17

Comparison with BuildingQA

Build2SPARQL is a large automatically generated corpus, whereas BuildingQA is a smaller expert-authored evaluation set. Their quality measures differ, and BuildingQA independently covers three of Build2SPARQL’s six pattern types.

Roles and pattern coverage

  • Build2SPARQL contains 6,136 queries across 201 buildings, versus BuildingQA’s 27 queries across four buildings; the scale difference reflects construction method, not quality.Build2SPARQL covers six pattern families, three of which appear in BuildingQA.Source: Comparison with BuildingQA · p. 17

Question quality

  • Build2SPARQL has lower lexical self-similarity than BuildingQA (0.170 vs. 0.238) and higher distinct-opening variety (0.740 vs. 0.641), while BuildingQA has higher mean BGP triple count (5.7 vs. 3.9).The passage cautions that the corpus differences make these figures unsuitable as a ranking of the resources.Source: Table 12 · p. 18, Comparison with BuildingQA · p. 18

Overlap in pattern types

  • BuildingQA’s expert-authored queries independently include UNION, branching, and linear patterns, but its 27-query sample cannot assess OPTIONAL or attribute-filtered patterns.Its authors attribute the absence of aggregation to a workflow convention: operators aggregate extracted tabular data rather than within SPARQL.Source: Comparison with BuildingQA · p. 18

Comparison with Automated and Prior Building-Domain Corpora

Build2SPARQL’s contribution is not the largest corpus, but a combination of structural breadth, multi-ontology grounding, source-building diversity, and register-differentiated phrasing. The comparison also notes that executable-query validation is not unique and that temporal and threshold reasoning remain outside its scope.

Comparison with Automated and Prior Building-Domain Corpora

  • Build2SPARQL combines broader structural patterns, Brick and ASHRAE 223P grounding, 201 buildings, and five register-differentiated questions per query.Its inventory includes branching, UNION, OPTIONAL, grouped aggregation, and unit/quantity attribute filtering; exclusion patterns and post-aggregation filters remain outside it.Source: Comparison with Automated and Prior… · p. 19
  • Build2SPARQL is smaller than OntoSage’s roughly 120,000 pairs and BuildingGPT and BuildingGPT2’s 18,555 distinct queries.Its distinguishing contribution is the combination of properties that none of the compared resources combines.Source: Comparison with Automated and Prior… · p. 19
  • Executable-query validation is not unique to Build2SPARQL, and temporal and threshold reasoning remain outside its static-structure inventory.The comparison identifies breadth across patterns, ontologies, buildings, and phrasing as Build2SPARQL’s contribution.Source: Comparison with Automated and Prior… · p. 19

Research Applications

The corpus supports text-to-SPARQL research, graph-completeness assessment, and controlled cross-building comparisons, with a reference split designed to prevent building-specific identifier leakage. Its use is bounded by pattern imbalance, top-down template coverage, and measurable differences between synthetic and real buildings.

Research Applications

  • Identical pattern templates across 201 buildings enable controlled comparisons of queryability across building types, ontology versions, and data quality.The corpus also supports model development and evaluation, while absent classes and relationship types can indicate under-populated graph elements.Source: Research Applications · pp. 19–20

Reference Evaluation Protocol

  • Query-level splits leak building-specific equipment identifiers; the reference split therefore partitions at the building level.It assigns 173 buildings to retrieval and 28 to test, including 15 held-out hospital buildings as a type-transfer probe and 13 randomly sampled real buildings.Source: Reference Evaluation Protocol · p. 20, Table 14 · p. 20

Limitations

  • 81.6% of the corpus consists of linear chains, which can skew raw-distribution training toward simple retrieval.The pattern-type field supports balanced or reweighted subsets, and per-pattern metrics should be reported.Source: Limitations · p. 20
  • The six-template, top-down benchmark excludes questions whose SPARQL falls outside its inventory, including temporal, threshold, and cross-building queries.Practitioners should treat it as a lower bound on real-world difficulty rather than an upper bound.Source: Limitations · pp. 20–21
  • Held-out synthetic buildings score 67–74% Exact Match F1, versus 43–55% for held-out real buildings, a roughly 20-point gap.The authors recommend reporting real-only held-out results and treating synthetic buildings as controlled structural diversity rather than a proxy for real-deployment difficulty.Source: Limitations · p. 21
  • The five questions per query are LLM-generated, and the authors note that larger-scale human rating would strengthen confidence in their quality.The two-rater validation found high semantic fidelity and naturalness on a stratified sample.Source: Limitations · p. 21

Conclusion

Build2SPARQL separates query correctness from linguistic generation: graph-traversal code produces and validates SPARQL, and LLMs generate questions from verified queries. The resulting benchmark supports retrieval-augmented evaluation, while its inventory and ontology scope leave clear directions for extension.

Conclusion

  • Graph-traversal code mines and validates SPARQL against source graphs, while LLMs generate natural-language questions only after queries are verified.This separation makes the KG the correctness oracle and guarantees queries are executable against their source buildings.Source: Conclusion · p. 21
  • Across 201 building KGs, six pattern families yield 6,136 executable queries and 30,680 questions spanning five vocabulary registers.The buildings comprise 81 real and 120 BuildGraph synthetic graphs.Source: Conclusion · p. 21
  • 56–65% Exact Match F1 with three-shot retrieval across three open-weight LLMs, compared with substantially lower zero-shot baselines.Qwen 2.5 32B reaches Row Matching F1 up to 0.66 with three-shot retrieval.Source: Conclusion · p. 22
  • FILTER NOT EXISTS and HAVING are direct extension targets, respectively enabling direct absence diagnostics and operational thresholds in SPARQL.HAVING requires a principled threshold-selection strategy.Source: Conclusion · p. 22
  • Extending beyond Brick requires adapting pattern mining and triple-walking validation to each ontology’s class hierarchy and property vocabulary.The paper identifies ASHRAE 223P, BOT, and IFC-OWL as schemas relevant to a unified multi-ontology corpus.Source: Conclusion · p. 22
  • A lightweight human annotation loop could flag questions with weak phrasing variety or register fidelity and trigger targeted regeneration.Pipeline version stamping would track which entries have been re-reviewed.Source: Conclusion · p. 22

Appendix I: Human Validation Protocol

The appendix describes a two-rater protocol for evaluating question fidelity, naturalness, and operational plausibility, including a Q4-specific grounding rule and adjudication procedure. The dataset and reference split are available through the Zenodo deposit, while source graphs are linked but not redistributed.

Appendix I: Human Validation Protocol

  • 300 questions from 60 stratified queries were sampled, with one question per register for each query.The sample covered all six pattern types, with minimum cell sizes for rarer types.Source: Appendix I · p. 22
  • Two raters independently coded the sample using a shared guide and ontology, query-structure, and register primer.They jointly coded the first three queries, reconciled differences, then independently coded the remaining items.Source: Appendix I · p. 23
  • For Q4, a question passes fidelity when its named equipment instance is in the query result set and the relationship matches.A condition absent from the query is borderline; an instance outside the result set or an incorrect relationship fails.Source: Appendix I · p. 23
  • After adjudication, pass rates were 98.8% for semantic fidelity, 98.8% for naturalness, and 84.0% for operational plausibility.The protocol scored Y as 1, B as 0.5, and N as 0; disagreements were resolved by consensus using the codebook.Source: Appendix I · p. 23, Table 16 · p. 24

Data Availability Statement

  • The Zenodo deposit includes NL/SPARQL pairs and the reference split, but not the source building graphs.Links to the original graph repositories are included.Source: Data Availability Statement · p. 24

Related papers

Loading 2610.00224v1…