Source-linked AI summary
sheval: An RDF data shapes evaluation tool and test-suite for recursive shapes
Shqiponja Ahmetaj, Iovka Boneva, Jan Hidders, Maxime Jakubowski, Jose-Emilio Labra-Gayo, Wim Martens, Filip Murlak, Cem Okulmus, Ognjen Savković, Mantas Šimkus, Dominik Tomaszuk
TL;DR
Recursive-shape semantics are specified in ShEx but left to SHACL implementations, which behave differently. The paper presents sheval, a tool and test suite for comparing engines; its tests find ShEx validators consistent with GFP while SHACL behavior is less uniform. The tests provide behavioral evidence, not proof that a validator follows a particular semantics.
Problem
Method
Results
The three ShEx validators were consistent with GFP, while SHACL validators showed less uniform behavior, including outcomes not consistent with the tested candidate semantics.
Source: §6 · pp. 9–10Takeaways & Limitations
sheval provides a framework for comparing engines against expectations and identifying inconsistencies in recursive-shape validation.
Source: §8 · p. 11The tests do not prove that validators follow a particular semantics; establishing that requires knowledge of and formal study of each system's algorithm.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Two different languages have been developed to validate RDF data based on the concept of a shape: ShEx and SHACL. In each language it is possible to define a shape that refers to itself, which is called a recursive shape. While in the case of ShEx, the semantics of recursive shapes is well defined and is part of the specification, in the case of SHACL, the semantics of recursive shapes is left to the implementation of the different SHACL engines. Consequently, the different SHACL engines show different behaviours when confronted with recursive shapes. In this paper we present sheval: an evaluation framework consisting of a tool and a test suite that can be used to compare the behaviour of different shapes technologies when confronted with recursive definitions. The tool has been used to evaluate and understand the differences in the implementation of recursive shapes in ShEx and SHACL. It provides a framework for testing and comparing the behaviour of different shape engines, helping to identify inconsistencies and potential issues, and providing a basis for further research and development in the field of shape-based validation of RDF data.
1. Introduction
ShEx specifies recursive-shape semantics, whereas SHACL leaves them to individual engines, which can behave inconsistently. The paper presents sheval and an extended test suite for comparing these technologies.
1. Introduction
- ShEx defines recursive shapes using stratified negation, while SHACL leaves their semantics to individual engines and lacks a definitive W3C solution.Different SHACL engines can consequently behave inconsistently when validating recursive shapes.Source: §1 · pp. 1–2
- sheval helps practitioners and researchers compare validators on recursive-shape cases to identify inconsistencies and potential issues.The project is open source and can be run in Docker to support reproducibility.Source: §1 · p. 2
- The paper presents an extended sheval test suite, including non-recursive controls and LFP/GFP duality tests, alongside a deeper analysis of results.The tests address recursive definitions and combinations of recursion with negation.Source: §1 · p. 2
2. Simple Shape Language and Recursive shapes
The section introduces a simple shape language for describing recursive shapes and explains how shape assignments and competing semantics determine validation outcomes.
2. Simple Shape Language and Recursive shapes
- The simple language represents shapes with constants, conditions, references, Boolean operators, and existential or universal predicate constraints.A catalog maps shape names to shapes, with all shapes in its image assumed to be defined.Source: §2 · pp. 2–3
- A shape assignment relates graph nodes to shape names, identifying which nodes are checked against which shapes and helping define recursive validation semantics.ShEx uses shape maps, while SHACL can use target declarations.Source: §2 · p. 3
- Recursive shapes can admit multiple valid assignments for the same graph and schema.The example gives three assignments: all nodes assigned, only two nodes assigned, or none assigned.Source: §2 · p. 3
- LFP assumes nonconformance unless constraints are met, GFP assumes conformance unless constraints are violated, and SMS allows any self-supporting interpretation.SMS validators may check any possible assignment (brave) or all possible assignments (cautious); prior work reports expressive-power overlap between ShEx with GFP and SHACL with LFP.Source: §2 · pp. 3–4
3. ShEx, SHACL and recursive shapes
ShEx and SHACL use different validation interfaces, and recursive-shape outcomes vary across engines because SHACL does not prescribe their semantics.
3. ShEx, SHACL and recursive shapes
- ShEx validators take an RDF graph, schema, and shape map, while SHACL engines use a graph, shapes graph, and optional targets to produce a validation report.ShEx specifies GFP semantics for recursive shapes.Source: §3 · pp. 4–5
- For the same recursive example, TopQuadrant reports that :a and :b do not conform, while Apache Jena reports conforms=true.The Jena result contrasts with the violations shown in the preceding report.Source: §3 · p. 5
- Other SHACL engines reject recursive inputs, warn and crash during validation, or enter an infinite loop.The reported examples are rudof, pySHACL, and SHACL-S, respectively.Source: §3 · p. 5
- Because SHACL 1.0 leaves recursive semantics undefined, these behaviors do not break the recommendation but create interoperability problems and mismatches with user expectations.The paper introduces a tool and tests for comparing such behaviors across shape technologies.Source: §3 · pp. 5–6
4. sheval architecture
sheval separates common test orchestration and result analysis from engine-specific execution. Its manifest-driven suites describe cases as data, which the tool transforms into inputs for requested validators.
4. sheval architecture
- sheval isolates technology-specific invocation and output parsing behind a uniform Runner interface while implementing orchestration, classification, semantics comparison, and reporting once.The shared flow handles engine outputs and recognized failures without placing crosscutting control flow in each runner.Source: §4 · p. 6
- A manifest describes suites and test cases as data, including graphs, schemas, targets, and optional expected results for LFP, GFP, bSMS, or cSMS.Adding a test case or suite does not require code changes.Source: §4 · p. 6
- The orchestrator expands declared nodes and shapes, prepares SHACL target-node graphs or ShEx ShapeMaps, and dispatches inputs to the requested engines.The CLI also supports running a full suite, invoking a single technology, or checking a raw binary.Source: §4 · p. 6
- The semantics module compares engine results with each test's expected models and records agreement with at least one or all candidate models.Results can be exported as YAML, CSV, or LaTeX, and bundled Docker dependencies support reproducible comparisons.Source: §4 · pp. 6–7
5. Recursive shapes test suite
The test suite probes competing recursive-shape semantics with separation tests, feature tests, duality tests, and non-recursive controls. Its cases specify graphs, shape catalogs, selectors, and expected assignments under candidate semantics.
5. Recursive shapes test suite
- Each test specifies an RDF graph, shape catalog, selector, and expected assignments under LFP, GFP, or supported-model interpretations.An assignment is classified as green when the graph conforms and brown when it does not.Source: §5 · p. 7
- Four minimal separation tests distinguish LFP from GFP semantics, including cases with multiple supported-model assignments.The cases cover self-recursion, mutually recursive shapes, and negation.Source: §5 · p. 8
- Four reachability and safety tests apply dual shape names across scenarios on a shared graph to compare LFP and GFP assignments.The tests use reachability and its dual property, safety.Source: §5 · p. 8
- Feature tests cover non-stratified recursion, logically inconsistent assignments, and selection of a fresh node outside the graph.The proposed semantics does not support non-stratified catalogs; the fresh-node case can be accepted or rejected.Source: §5 · pp. 8–9
- Duality tests check the proposition relating LFP and GFP by pairing shapes that swap negation, conjunction, and universal quantification for their dual forms.The resulting shape assignments for LFP and GFP are complementary.Source: §5 · p. 9
- Two non-recursive control tests check expected behavior, with equivalent assignments under LFP, GFP, bSMS, and cSMS.One tests a basic non-recursive shape; the other expects non-conformance.Source: §5 · p. 9
6. Discussion
The results show agreement with GFP among the ShEx validators, while SHACL validator behavior varies by engine. The tests offer evidence about implementation behavior but do not prove which semantics an engine implements.
6. Discussion
- All three ShEx validators are consistent with GFP for recursive shapes and non-stratified catalogs, as prescribed by the official semantics.The results table reports outcomes for ShEx and SHACL validators on recursive-shape catalogs.Source: §6 · p. 9, Table 1 · p. 10
- SHACL behavior varies: pySHACL often returns conforms=true after detecting cycles, Jena SHACL continues validation consistently with GFP, and SHACL-TQ matches none of the four candidate semantics.pySHACL errors on example 2 and bsep3; SHACL-S fails those cases and has a timeout; SHACL-TQ may require closer algorithm and source-code analysis.Source: §6 · p. 9
- Rudof SHACL's selected GFP or LFP mode matches the semantics indicated by its flag.Rudof also has a flag for stopping on recursive shapes.Source: §6 · p. 10
- For non-stratified catalogs, ShEx-S and rudof correctly reject inputs, while other SHACL validators attempt validation and return non-conformance.Jena ShEx does not reject these inputs and behaves asymmetrically on two similar consistency tests.Source: §6 · p. 10
- In duality tests, ShEx validators are consistent with GFP, and SHACL validators also seem to follow GFP.The reported result is framed as an apparent semantic pattern.Source: §6 · p. 10
- The tests can reveal implementations that do not follow a given semantics, but cannot prove which semantics a validator follows without studying its algorithm formally.The results provide hints about implementation behavior.Source: §6 · p. 10
7. Related work
Prior work includes formal analysis of recursive shape semantics and benchmarks or test suites for shape validators. Existing suites cover recursion unevenly or do not examine its alternative semantics.
7. Related work
- This paper follows a formal analysis of recursive shape notions [3] and builds on earlier work seeking a common foundation for shape languages [11].The earlier common-foundation work did not include recursion.Source: §7 · p. 11
- The Webindex benchmark includes recursive shapes inspired by a real-world use case, but does not explore their different semantics [21].Source: §7 · p. 11
- The SHACL test suite lacks recursion tests, while the ShEx suite includes recursive shapes but examines only GFP semantics.The paper notes that the recursive suite could extend SHACL testing if future work addresses recursive validation.Source: §7 · p. 11
8. Conclusions
The paper presents sheval as a tool for evaluating shape engines and comparing their handling of recursive shapes. It identifies future work in adding engine implementations and extending test-suite coverage.
8. Conclusions
- sheval has been used to evaluate differences in how ShEx and SHACL implement recursive shapes and to compare engines against expected behavior.It can help identify inconsistencies and potential issues, and its reports can be serialized in YAML, CSV, or LaTeX.Source: §8 · p. 11
- The tool is customizable and generates detailed reports for later analysis.Reports can be serialized in YAML, CSV, or LaTeX.Source: §8 · p. 11
- Future work includes adding ShEx and SHACL implementations and test suites covering expressiveness, interoperability, performance, explainability, or usability.Source: §8 · p. 11
Declaration on Generative AI
The authors used ChatGPT, Copilot, and Claude Code for language and style editing, then reviewed and edited the content and retained responsibility for it.
Declaration on Generative AI
- The authors used ChatGPT, Copilot, and Claude Code for grammar, paraphrasing, and style improvements, then reviewed the content and accepted responsibility for it.Source: Declaration on Generative AI · p. 12