Source-linked AI summary
SPARQL-LLM: Real-Time SPARQL Query Generation from Natural Language Questions
Panayiotis Smeros, Vincent Emonet, Ruijie Wang, Ana-Claudia Sima, Tarcisio Mendes de Farias
TL;DR
Generating SPARQL from natural-language questions is challenging, while many LLM approaches emphasize accuracy and give less attention to runtime and cost. SPARQL-LLM uses lightweight endpoint metadata in a hybrid retrieval-and-refinement approach. The paper reports stronger challenge performance, faster generation, and deployment over real-world decentralized knowledge graphs.
Problem
SPARQL generation from natural language remains challenging, and many LLM approaches focus on accuracy while giving less attention to runtime and cost.
Source: §1 · p. 2Method
SPARQL-LLM uses lightweight metadata from SPARQL endpoints and combines retrieval-based query generation with iterative refinement.
Source: §1 · p. 2Results
SPARQL-LLM reports up to 59% improvement in F1 score over the challenge’s second-best system and up to 27× faster runtime, with its conclusion describing high accuracy, low latency, and minimal operational costs.
Source: §1 · p. 2, §5 · p. 17Takeaways & Limitations
The authors report that SPARQL-LLM is deployed in production to generate and execute complex queries against real-world decentralized knowledge graphs.
Source: §5 · p. 17The evaluation does not systematically cover other high- and low-resource languages, additional domains, the full SPARQL functionality, or example quality and noise.
Source: §5 · p. 18
Abstract
from arXiv · showhide
The advent of large language models is contributing to the emergence of novel approaches that promise to better tackle the challenge of generating structured queries, such as SPARQL queries, from natural language. However, these new approaches mostly focus on response accuracy while ignoring other evaluation criteria, such as runtime and cost to generate SPARQL queries. Consequently, they are often not production-ready or easy to deploy over real-world knowledge graphs with good accuracy. To mitigate these issues, in this paper, we describe and systematically evaluate SPARQL-LLM, an open-source and triplestore-agnostic approach, powered by lightweight metadata, that generates SPARQL queries from natural language text. First, we describe its architecture, which consists of dedicated components for metadata indexing, prompt building, and query generation and execution. Then, we evaluate it based on a state-of-the-art challenge with multilingual questions, and a collection of questions from three of the most prevalent knowledge graphs within the field of bioinformatics. Our results demonstrate a substantial improvement of up to 59% in F1 score over the second-best system participating in the challenge, adaptability to high-resource languages such as English, Spanish, and German, as well as ability to form complex bioinformatics queries. Furthermore, our results show that our system is up to 27x faster than the second-best system participating in the challenge, while costing a maximum of $0.01 per question, making it suitable for real-time, low-cost text-to-SPARQL applications. SPARQL-LLM is publicly released as an open-source project at https://github.com/sib-swiss/sparql-llm and is currently deployed over real-world decentralized knowledge graphs at https://www.expasy.org/chat.
1 Introduction
SPARQL-LLM targets the difficulty of formulating SPARQL over heterogeneous, complex knowledge graphs, especially in specialized domains. It combines retrieval and iterative query refinement in a modular system, reporting stronger challenge performance, multilingual adaptability, and low latency and cost.
1 Introduction
- SPARQL formulation is difficult because it requires knowledge of query syntax, access methods, data schemas, and connections across data, with added complexity in specialized domains.Domain experts may spend considerable time formulating, testing, and refining queries.Source: §1 · p. 2
- The system combines retrieval-based query generation with agent-based iterative refinement for query validation.Its modular architecture includes metadata indexing, prompt construction, and query generation and execution.Source: §1 · p. 2
- Up to 59% improvement in F1 score over the challenge’s second-best system is reported for SPARQL-LLM.The system also reports adaptability to English, Spanish, and German, and handling complex bioinformatics queries.Source: §1 · p. 2
- SPARQL-LLM is up to 27× faster than the challenge’s second-best system and requires as little as $0.01 per question.Source: §1 · p. 2
- SPARQL-LLM is released as open source and deployed over real-world decentralized bioinformatics knowledge graphs for experts and non-experts.Source: §1 · pp. 2–3
- The paper extends prior preliminary work with a detailed architecture, a new SPARQL Execution component, and a formal evaluation beyond a limited, non-standardized test set.Source: §1 · p. 3
2 Related Work
Prior KGQA work includes agent-based graph exploration and retrieval-based context construction, while SPARQL-LLM combines retrieval with iterative query validation. The paper situates its evaluation across general-knowledge and specialized bioinformatics questions.
2 Related Work
- Agent-based systems iteratively use tools to explore knowledge graphs, but can incur long runtimes and high token costs and struggle with complex query patterns.Schema patterns may need to be rediscovered for each question, and agents may fail to converge or require many iterations.Source: §2 · p. 3
- Related systems use retrieval to place data shapes and ontology fragments in prompts, or fine-tune autoregressive models for query generation.Source: §2 · p. 3
- SPARQL-LLM combines retrieval-based query generation with agent-based iterative refinement to validate queries, aiming for real-time generation with minimal LLM interactions.Source: §2 · p. 3
- The paper reports evaluating general-knowledge and specialized bioinformatics questions, whereas much prior evaluation targets encyclopedic questions or scholarly knowledge graphs.It characterizes SPARQL-LLM as the first approach evaluated on both general-knowledge and bioinformatics-domain questions.Source: §2 · p. 3
3 SPARQL-LLM Overview
The overview section describes the text-to-SPARQL approach and its system architecture, focusing on how context is retrieved or generated.
3 SPARQL-LLM Overview
3.1 Provided Context
SPARQL-LLM draws on endpoint query examples and a data-aware schema to construct prompts, with examples grounding user intent and compact ShEx shapes describing class-specific predicates and targets.
3.1 Provided Context
- SPARQL-LLM uses described SPARQL-query examples and a data-aware schema, potentially with frequency indicators, as sources for query construction.Source: §3.1 · p. 4
- Endpoint metadata can come from a complete but costly VoID generator or SPARQL-LLM’s faster-trade-off generator for changing RDF datasets.Source: §3.1 · p. 4
3.1.1 Query Examples.
- The authors favor real-world, human-crafted query examples because they capture end-user intent better than self-generated examples [18].Source: §3.1.1 · p. 4
3.1.2 Data-Aware Schema.
- ShEx shapes make each class self-contained and expose its available predicates and target classes or datatypes; their compactness reduces LLM tokens and inference costs.The authors leave systematic comparison with OWL and SHACL for future work.Source: §3.1.2 · pp. 4–5
3.2 System Architecture
SPARQL-LLM indexes endpoint examples and schema, retrieves relevant context for a question, then generates, validates, executes, and interprets a query.
3.2 System Architecture
- The system’s data flow retrieves examples and schema classes for a question, builds a generation prompt, validates and corrects the query, executes it, and interprets the response.Source: §3.2 · p. 6
3.2.1 Indexing Component.
- At deployment, the indexing component embeds endpoint examples and schema classes and stores them in a vector database for later semantic retrieval.Source: §3.2.1 · p. 6
3.2.2 Prompt Building Component.
- Prompt building splits complex questions into standalone sub-questions and extracts high-level concepts for retrieval; atomic questions are not decomposed.Source: §3.2.2 · p. 6
3.2.3 SPARQL Generation Component.
- The generation component checks queries syntactically and structurally against schema, then uses parser or execution errors to refine them iteratively, typically within three iterations.It omits full ontology reasoning and cannot guarantee semantic correctness.Source: §3.2.3 · pp. 6–7
3.2.4 SPARQL Execution Component.
- After validation, queries run against the endpoint, and the LLM explains results; federated endpoint results are combined before interpretation.Source: §3.2.4 · p. 7
4 Evaluation
The evaluation covers challenge participation and bioinformatics questions, using execution-based F1 to assess generated-query results and examining the context supplied to the system.
4 Evaluation
- The evaluation includes post hoc participation in TEXT2SPARQL 2025, questions from three bioinformatics knowledge graphs, and live participation in the 2026 challenge.Source: §4 · p. 7
- The reported TREC F1 is execution-based: it measures overlap between generated-query and ground-truth result sets, flattening multi-variable results and averaging query scores without weighting.Source: §4 · pp. 7–8
4.1 Post Hoc KGQA Evaluation
- TEXT2SPARQL 2025 tests DBpedia with 100 questions each in English and Spanish and an undocumented corporate KG with 50 English questions; experiments ran three times with 95% confidence intervals.Source: §4.1 · p. 8
4.1.1 Provided Context.
- DBpedia examples come from LC-QuAD and QALD-9+, and challenge DBpedia queries mostly contain one to three triple patterns, similar to those examples.Source: §4.1.1 · p. 8, Figure 3 · p. 8
System Analysis
This section analyzes SPARQL-LLM’s performance across the TEXT2SPARQL challenges and BioKGQA, alongside cost and hyperparameter trade-offs, error sources, and bioinformatics question complexity and setup.
4.1.2 Systems.
- The challenge evaluation compared GPT-4o, GPT-4o-mini, and GPT-oss-120b variants whose knowledge cutoffs preceded the challenge, reducing the chance of prior dataset knowledge.Only GPT-based variants were evaluated because challenge participants used GPT variants; other flagship LLMs were evaluated separately.Source: §4.1.2 · pp. 9–10
4.1.3 Results.
- 24% higher F1 Score on DBpedia (EN) and DBpedia (ES) was achieved by SPARQL-LLM_lg, while SPARQL-LLM_os beat challenge winners on all three subtasks.SPARQL-LLM_os achieved the best Corporate performance; SPARQL-LLM performed better in F1 Score across all three subtasks.Source: Figure 4 · p. 10, §4.1.3 · p. 10
4.1.4 Cost Analysis.
- 2.2s median runtime for SPARQL-LLM_sm on Corporate compared with 59.6s for ARUQULA; average SPARQL-LLM cost was at most $0.0113 per question.The analysis evaluates runtime and input/output token use as cost dimensions; it notes that SPARQL-LLM_sm was slightly faster on average than SPARQL-LLM_lg, while SPARQL-LLM_os was slowest.Source: §4.1.4 · pp. 10–11, Figure 5 · p. 11
4.1.5 System Analysis.
- For DBpedia, the best performance used the 25% most frequent schema, while more schema improved Corporate results; multilingual embeddings performed better on DBpedia (ES).Embedding size did not significantly affect performance, and schema fraction had little effect on DBpedia accuracy while reducing context and per-question cost.Source: §4.1.5 · pp. 11–13, Figure 6 · p. 12
- Performance plateaued after 10 examples, peaked with examples of up to two triple patterns, and declined for more complex Corporate examples; ablations ranked examples, class schema, then validation as essential.Five to ten relevant, ideally non-overlapping examples were considered sufficient; all tested LLMs performed similarly, with GPT-4o slightly superior, and results were not intended as a comprehensive benchmark. Removing examples or validation reduced performance; class information mattered for Corporate but schema had little effect on DBpedia.Source: Figure 6 · p. 12, §4.1.5 · p. 13
4.1.6 Error Analysis.
- Seven error categories included namespace and alternative-predicate errors from DBpedia’s redundant schema, redundant aggregate-variable projection, wrong encoding or query type, and empty queries; no top-k errors were observed.Redundant projection can lower F1 despite semantic equivalence because results are flattened when multiple variables are projected.Source: §4.1.6 · p. 14
- 11 erroneous reference queries were found; examples showed generated queries correcting class bindings, COUNT/DISTINCT formulation, and the Apollo 11 landing predicate.In the population example, neither the reference nor generated answer was correct because of the challenge data dump, although the generated query bound the entity to a Country class.Source: §4.1.6 · pp. 14–15
4.2 BioKGQA Evaluation.
- BioKGQA evaluates Uniprot, Cellosaurus, and Bgee using 3-fold cross-validation; their query sets contain 123, 73, and 28 examples, respectively, with some queries having 10× more triple patterns.The knowledge graphs have public SPARQL endpoints and lightweight metadata, and the queries are more complex than those in TEXT2SPARQL 2025.Source: §4.2 · p. 15, Figure 7 · p. 15
4.2 BioKGQA Evaluation
- BioKGQA performance dipped slightly relative to standard KGQA, but SPARQL-LLM_lg remained the system’s most effective variant.The authors characterize the results as encouraging given the difficulty of reconstructing complex queries, even for domain experts.Source: §4.2 · p. 16
- SPARQL-LLM’s UniProt F1 of 0.414 is not directly comparable to GRASP’s 0.207 because query samples and data dumps differ; it uses the live endpoint and lightweight metadata without on-the-fly searches.Source: §4.2 · p. 16
4.3 Live KGQA Evaluation
- SPARQL-LLM improved F1 by 17% on DBpedia (English), 59% on DBpedia (Spanish), 32% on Corporate (English), and 30% on Corporate (German).The challenge covered both subtasks and all three languages.Source: Table 3 · p. 17, §4.3 · p. 17
- SPARQL-LLM outperformed other systems on Precision, Recall, and NDCG, while total LLM-call costs for the challenge were less than $3.Source: §4.3 · p. 17
5 Conclusion
SPARQL-LLM is presented as a triplestore-agnostic system that combines endpoint metadata with contextual prompts and is deployed on real-world decentralized knowledge graphs. The authors identify evaluation boundaries and several directions for future work.
5 Conclusion
- SPARQL-LLM combines lightweight endpoint metadata with contextual prompts and is reported to outperform iterative graph-traversal and prompt-refinement approaches in accuracy, latency, and operational cost.The authors report production deployment for generating and executing complex queries against real-world decentralized knowledge graphs.Source: §5 · p. 17
- The evaluation is limited by available KGQA benchmarks and does not systematically cover more languages, domains, SPARQL functionality, or example quality and relevance.The authors advocate more diverse, continuously evolving benchmarks.Source: §5 · p. 18
- The authors cannot fully isolate the effect of provided context from LLMs’ prior knowledge because the evaluated high-end models use closed training corpora.Models without prior exposure to SPARQL or programming languages would be ideal for this experiment, but are currently infeasible to use.Source: §5 · p. 18
- Future work includes handling more complex federated queries, testing schema representations and context-retrieval strategies, studying user experience, and extending integrations and data-access workflows.Proposed extensions include MCP service deployment and query generation for other access methods such as FastAPI.Source: §5 · p. 18
Declaration on Generative AI
The authors used generative AI tools for language and style edits, then reviewed and edited the content and retained responsibility for it.
Declaration on Generative AI
- The authors used Claude and GPT for grammar, spelling, paraphrasing, and stylistic edits, then reviewed and edited the content and took full responsibility for it.Source: Declaration on Generative AI · p. 18