Source-linked AI summary
Strengthening LargeRDFBench for Interoperable Federated SPARQL Evaluation
Bryan-Elliott Tam, Muhammad Saleem, Ruben Taelman
TL;DR
LargeRDFBench’s non-conformant datasets and ad hoc expected-results format limit reproducible, interoperable federated SPARQL evaluation. The paper repairs and standardizes the benchmark, validates it independently, and compares ASK- and COUNT-based FedX source selection, while identifying unresolved semantic and hosting-engine effects.
Problem
LargeRDFBench contains datasets that strict RDF parsers cannot ingest and expected results distributed in a custom format, limiting benchmark interoperability and reproducibility.
Method
The paper applies a reproducible cleaning pipeline, produces standards-conformant serializations, re-encodes expected results in W3C SPARQL JSON, and validates them end-to-end.
Results
The modernized benchmark supports strict RDF parsing and standard machine-verifiable results; ASK is faster on the median, while per-query ratios range from 0.15 to 123.62.
Takeaways & Limitations
Benchmark artifacts can be aligned with RDF standards to broaden reproducible comparisons, while automatic source-selection results require clearer execution definitions.
Takeaways & Limitations
FedQPL leaves real SPARQL execution under automatic source selection ambiguous because it uses set semantics while SPARQL uses bag semantics.
Abstract
from arXiv · showhide
LargeRDFBench is one of the most comprehensive benchmarks for evaluating federated SPARQL query engines, combining real, interlinked datasets with a rich query suite that has made it a reference point for the community. Evaluations of federated engines are published by comparing engine results against the benchmark's expected results, so those expected results must themselves be reproducible. Moreover, several of its data dumps violate the RDF specifications, so only engines that parse RDF leniently can host them, and its expected results are distributed in an ad hoc format. We identify, categorize and repair these data-quality issues with a reproducible cleaning pipeline, producing standards-conformant serializations of every affected dataset. Furthermore, we re-encode the benchmark's expected results in the W3C SPARQL 1.1 Query Results JSON Format and correct their discrepancies. Every dataset now parses under strict RDF parsers, and the expected results are machine-verifiable through a standard format, extending the benchmark's reach to the full range of conformant engines while staying faithful to the original data. Reproducing the expected results end-to-end with an independent implementation uncovers corruption in the published reference, and discrepancies between our results and the original ones, some not trivial to resolve, others open questions. We further perform a preliminary comparison, not previously explored, of ASK- and COUNT-based source selection in the FedX algorithm. This work strengthens an already valuable community resource by aligning its artifacts with the RDF standards, broadening the set of engines that can be fairly and reproducibly compared. We also raise the question of how the results of federated queries under automatic source selection can be made reproducible.
1. Introduction
LargeRDFBench supports federated SPARQL evaluation over real, interlinked datasets, but standards violations and ad hoc expected results undermine interoperability and reproducibility. The paper repairs these artifacts and validates them end-to-end, exposing further reproducibility challenges.
- Federated SPARQL combines facts from separate datasets across multiple endpoints without centralizing their data.
- Benchmark interoperability requires independent implementations to ingest both prescribed data and query scenarios.
- Published RDF-quality problems motivate repairs because syntactic validity is a recognized data-quality dimension.
- LargeRDFBench contains 13 real datasets, 32 queries, and expected results, but several datasets reject strict RDF parsers.
- Hosting-engine behavior and unresolved FedQPL semantics can change federated answers independently of the client-side algorithm.
- The paper produces a standards-conformant benchmark, validates expected results against source datasets, and corrects discrepancies with reproducible verification scripts.
2. Modernization Process
The modernization process deterministically cleans LargeRDFBench datasets, serializes them as valid N-Triples, and re-encodes aligned expected results in a W3C standard format. Repairs target RDF syntax violations and selected tooling-compatibility defects.
- Modernization Process: The open-source GNU make pipeline downloads, cleans, serializes, validates, and records per-dataset repairs reproducibly.
- Modernization Process: Expected results are re-encoded in the W3C SPARQL 1.1 Query Results JSON Format and repaired to remain aligned with the datasets.
- Taxonomy of the Repairs: The pipeline groups standards violations into three classes and applies deterministic repairs, with repair frequencies aggregated in Table 1.
- Taxonomy of the Repairs: Invalid IRIs are repaired through percent-encoding, base-IRI resolution, and removal of leading or trailing spaces.
- Taxonomy of the Repairs: Strictly invalid multiline literals are rejoined with escaped newline characters for N-Triples serialization.
- Taxonomy of the Repairs: Additional compatibility repairs replace NUL bytes and convert single-quoted RDF/XML attributes to double quotes.
- Taxonomy of the Repairs: Malformed language tags are reduced to their primary language subtags, such as fr_1793 becoming fr.
3. Experiment
The experiment validates the modernized benchmark end-to-end and compares ASK- and COUNT-based FedX source selection on a QLever-hosted federation. ASK is faster at the median, but per-query performance varies widely and reproducibility remains semantically and operationally constrained.
- Experiment: The experiment validates benchmark integrity and compares FedX ASK- and COUNT-based source selection using planning and execution times.
- Experiment: The evaluation used 14 machines, QLever at all endpoints, Comunica’s FedX client, simple and complex queries, and 30-minute query limits.
- Experiment: Table 2 reports median, minimum, and maximum planning and execution times plus the per-query ASK/COUNT ratio.
- Reproducibility discrepancies: FedQPL’s set semantics do not specify how real SPARQL execution under automatic source selection reconciles with SPARQL bag semantics.
- ASK versus COUNT: ASK is faster on the median, but the per-query ASK/COUNT execution ratio ranges from 0.15 to 123.62.
4. Conclusion
The repaired benchmark enables reproducible cross-engine comparison, while validation reveals unresolved semantic obstacles for federated queries with automatic source selection.
- Every conformant engine can now host the repaired benchmark data, supporting reproducible cross-engine comparison.
- End-to-end reproduction exposed three obstacles: string ingestion differences, materialized entailments, and missing formal semantics for automatic source selection.Examples include trailing-space handling and datatype collapsing that can alter solution multiplicities.
- FedQPL executes source-selection plans over sets of solution mappings, whereas SPARQL uses bag semantics, leaving automatic federated query results formally underspecified.
- Extending FedQPL or defining an alternative bag-semantics formalism is identified as an open problem for reproducible federated evaluation.
- ASK- and COUNT-based source-selection costs vary across queries and depend on how hosting engines evaluate them.The paper leaves systematic multi-engine evaluation for future work.
Declaration on Generative AI
During preparation, the authors used LLM technologies for drafting content and code generation, then reviewed and edited the resulting material.
- The authors used LLM technologies for drafting content and code generation during preparation.
- They reviewed and edited the generated material as needed and retained full responsibility for the publication’s content.