Source-linked AI summary
Towards a Question Answering System over the Semantic Web
Dennis Diefenbach, Andreas Both, Kamal Singh, Pierre Maret
TL;DR
QA over knowledge bases commonly focuses on one language and one KB, while multilingual, KB-agnostic coverage remains limited. The paper presents an approach for multilingual QA across KBs and reports evaluation across benchmarks, alongside an integrated service for users. Its conclusion is a first approximation toward QA over the Semantic Web.
Problem
Current QA research does not address multilingual, KB-agnostic QA for both full and keyword questions, while many approaches depend on language-specific tools or KB assumptions.
Source: §1 · p. 1Method
The approach uses word semantics to interpret questions, constructs candidate SPARQL queries, and is designed to work across languages and KBs.
Source: §3 · p. 3, §3.2 · p. 4, §3.6 · p. 6Results
The authors report extensive evaluation across multiple benchmarks, outperforming many existing systems, querying multiple KBs, and scaling to very large KBs such as DBpedia.
Source: §7 · p. 13Takeaways & Limitations
The integrated services let end-users query multiple KBs in multiple languages through a unified interface, a step toward QA over the Semantic Web.
Source: §7 · p. 13WebQuestions reflects a reification model that was not its original goal, and more than 60% of systems benchmarked on it are tailored to that model.
Source: §5.2.1 · p. 12
Abstract
from arXiv · showhide
Thanks to the development of the Semantic Web, a lot of new structured data has become available on the Web in the form of knowledge bases (KBs). Making this valuable data accessible and usable for end-users is one of the main goals of Question Answering (QA) over KBs. Most current QA systems query one KB, in one language (namely English). The existing approaches are not designed to be easily adaptable to new KBs and languages. We first introduce a new approach for translating natural language questions to SPARQL queries. It is able to query several KBs simultaneously, in different languages, and can easily be ported to other KBs and languages. In our evaluation, the impact of our approach is proven using 5 different well-known and large KBs: Wikidata, DBpedia, MusicBrainz, DBLP and Freebase as well as 5 different languages namely English, German, French, Italian and Spanish. Second, we show how we integrated our approach, to make it easily accessible by the research community and by end-users. To summarize, we provided a conceptional solution for multilingual, KB-agnostic Question Answering over the Semantic Web. The provided first approximation validates this concept.
1. Introduction
The introduction identifies multilingual, KB-agnostic QA for both full and keyword questions as an unaddressed challenge. It presents an algorithm intended to address existing portability and scalability drawbacks while competing with other approaches in F-measure.
1. Introduction
- Current research does not address multilingual, KB-agnostic QA for both full and keyword questions.The paper frames this as a challenge in translating natural-language questions into SPARQL queries over knowledge bases.Source: §1 · p. 1
- Existing approaches can be difficult to port because they rely on language-specific tools, assume particular KB structures, or require costly training data.The introduction also notes that some approaches have not been shown to scale well.Source: §1 · p. 1
- The paper presents an algorithm addressing these drawbacks and says it can compete with existing approaches in F-measure.The introduction previews evaluation across different benchmarks.Source: §1 · p. 1
2. Related work
Related work is concentrated on English QA over one knowledge base, with multilinguality and portability rarely addressed. The paper situates its system within modular QA frameworks by integrating it as a Qanary component.
2. Related work
- WebQuestions and SimpleQuestions are large benchmarks, while QALD versions include more complex questions but contain only 100–450 questions.WebQuestions has 5,810 questions and SimpleQuestions has 108,442; the latter are answerable using a single binary relation.Source: §2 · p. 2
- Most QA systems work only in English and over one KB; fewer than 10% were applied to multiple languages and 5% to multiple KBs.The passage attributes this pattern to heavy use of NLP tools or natural-language features.Source: §2 · p. 2
- The system is integrated as the Qanary QA component WDAqua-core1, using a framework with off-the-shelf services and configurable components.The paper describes modularity as a way to develop QA systems and expose services to end-users.Source: §2 · p. 2
3. Approach for QA over Knowledge Bases
The approach interprets question words through the semantics encoded in a knowledge base, generating and ranking possible SPARQL queries rather than relying on question syntax. Its four stages are designed to work across languages and knowledge bases.
3. Approach for QA over Knowledge Bases
- The approach uses question-word semantics and knowledge-base semantics to infer intent, with question syntax playing a smaller role.It comprises question expansion, query construction, query ranking, and response decision, without assumptions about language or knowledge base.Source: §3 · pp. 2–3
3.1. Expansion
- Expansion finds entities, properties, and classes whose lexicalizations match question n-grams, while excluding matches for stop words.The stop-word exclusion is motivated by their limited semantics and by reducing time in the next step.Source: §3.1 · p. 3
3.2. Query construction
- Query construction searches to depth 2 from expanded IRIs and builds non-empty candidate queries that represent possible question intentions.It also adds direct-entity query patterns for questions such as “Who is Marie Curie?”.Source: §3.2 · pp. 3–4
3.3. Ranking
- Candidates are ranked using five features, including resource relevance and query size; weights are manual without training data or learned against F-measure when labeled data is available.The exemplary implementation uses RankLib with Coordinate Ascent, then executes the top-ranked query; syntactic features are not used.Source: §3.3 · p. 5
3.4. Answer Decision
- A logistic-regression confidence score for the top-ranked query determines whether the system answers or refuses the question.It answers only when the score meets threshold θ2.Source: §3.4 · p. 5
3.5. Multiple KBs
- For multiple knowledge bases, expansion considers their labels and construction treats them as one graph with unconnected components; ranking and answer decision remain unchanged..Source: §3.5 · p. 5
3.6. Discussion
- The approach can bridge implicit relations and jointly disambiguate entities and relations, while accommodating differences in how knowledge is encoded across knowledge bases.The German-mathematicians example links mathematicians to Germany through possible properties such as occupation and nationality.Source: §3.6 · pp. 5–6
- Language adaptation involves stop-word removal and stemming, and generated query lists can support ranker refinement; the system also supports malformed and keyword questions without NLP tools.The exemplary implementation identifies relations using a dictionary, which the authors describe as a disadvantage and a possible target for improvement.Source: §3.6 · pp. 6–7
4. Fast candidate generation
This section describes efficient SPARQL candidate generation by representing the KB as a directed labeled graph, computing distances from resources with bidirectional breadth-first search, and using those distances to construct connected triple patterns recursively.
4. Fast candidate generation
- The KB is modeled as a directed labeled graph, with vertices, edges, and labels assigned to edges.The graph is traversed in both directions for distance computation.Source: §4 · p. 7
- Breadth-first search from each resource in R computes pairwise distances, assigning signed values when traversing edges in reverse.These distances support later triple-pattern construction.Source: §4 · p. 7
- HDT’s adjacency-list representation supports breadth-first search, and the authors cite its low disk and memory footprint as an additional advantage.They give an approximate example of a 100 GB RDF dump compressed to a 10 GB HDT file.Source: §4 · p. 7
- The collected distances enable fast recursive construction of connected triple patterns from resources in R, up to a maximal K triples.The algorithm’s stated output is all connected triple patterns meeting that bound.Source: §4 · p. 8
5 Function generate (L,k)
The recursive generator builds connected triple patterns up to a maximum size, using distance-pattern checks to control its search. After construction, a question’s opening regex determines whether the system generates a SELECT or ASK query.
5 Function generate (L,k)
- The generator tests distance patterns and updates Boolean flags, proceeding when at least one flag remains true.The authors state that the “if not” conditions are often not fulfilled, which guarantees the process’s speed.Source: §5 · p. 8
- The recursive function advances k and adds connected triple patterns to L, which groups patterns by their number of triples, up to maximal K.Once patterns are built, a variable in subject or object position can be chosen as the projection variable.Source: §5 · p. 8
- Regex expressions over the beginning of the question determine whether the generated query is SELECT or ASK.The passage notes that the system may also need to decide whether a COUNT is needed.Source: §5 · p. 8
5. Evaluation
The evaluation tests portability and robustness across varied benchmarks, five knowledge bases, and five languages. Results cover multilingual performance, multiple-KB querying, training-data requirements, and runtime.
5.2. Experiments
- The evaluation spans benchmarks with different demands, including simple triple-pattern questions, complex questions, multilingual questions, and keyword questions.The authors provide first baselines for LC-QuAD and WDAquaCore0Questions.Source: §5 · p. 9, Table 6 · p. 11, §5.2.1 · p. 12
5.1. Stop Words and lexicalizations
- Lexicalizations were gathered from KB labels and links, with additional DBpedia property phrases extracted from abstracts; one hour of manual work aligned MusicBrainz and DBLP properties with Wikidata.For SimpleQuestions, property lexicalizations were manually collected from the first 100 training questions, rather than the remaining training and validation questions.Source: §5.1 · p. 9
5.2.1. Benchmarks
- On English QALD benchmarks, the approach outperformed 90% of proposed systems, but not Xser or UTQA, which used additional training data.For QALD-6, reported error sources included lexical gaps (40%) and wrong ranking (28%); keyword questions performed worse than full natural-language questions.Source: §5.2.1 · p. 10, Table 3 · p. 11
- On Wikidata’s QALD-7 task 4 training dataset, English performed best and Italian worst, which the authors relate mostly to Italian’s poorer lexicalization coverage.Except for English, keyword questions were easier than full natural-language questions; the approach outperformed the presented system on this benchmark.Source: §5.2.1 · p. 10, Table 4 · p. 11
- On SimpleQuestions, performance was 14% below state of the art using 100 training questions and one hour of manual property lexicalization work.The 108,442-question benchmark can be solved with one triple pattern; the authors contrast their training effort with generating an 80,000-question dataset.Source: §5.2.1 · p. 10, Table 5 · p. 11
- Querying five KBs on QALD-6 yielded better results than DBpedia alone, with higher recall and lower precision; WDAqua-core1 selected DBpedia 53 times, Wikidata 39 times, and other KBs 8 times.The authors note that answering questions took under two seconds across benchmarks except when all KBs were queried simultaneously.Source: Table 7 · p. 11, §5.2.1 · p. 12, §5.2.2 · p. 12
- The authors did not evaluate WebQuestions because many benchmarked systems are tailored to its Freebase reification model, whereas Semantic Web KBs commonly use binary statements and have varied reification models.They note that more than 60% of systems benchmarked on WebQuestions are tailored to that model.Source: §5.2.1 · p. 12
5.2.2. Setting
- The experiments ran on a 4-core, 16-GB-RAM virtual machine hosting the indexes and triple stores; the original data dumps totalled 336 GB.All-KB querying was the exception to the under-two-second response time reported across benchmarks.Source: §5.2.2 · p. 12
6. Provided Services for Multilingual and Multi-KB QA
WDAqua-core1 was integrated into Qanary and connected with Trill to provide an end-user-facing QA service. Trill was extended to support five KBs and make adding further KBs easier.
6. Provided Services for Multilingual and Multi-KB QA
- WDAqua-core1 was integrated into Qanary, exposing RESTful interfaces and enabling combinations with speech recognition and language detection components.The integration also enables reuse of Trill, a reusable QA front end.Source: §6 · p. 12
- Trill was extended from DBpedia and Wikidata to also support MusicBrainz, DBLP, and Freebase, with new-KB support mainly requiring an adapted SPARQL query.The extension lets users select multiple KBs simultaneously.Source: §6 · p. 13
- SPARQLToUser and LinkSUM were adapted to support the five KBs and five languages.Figure 4 shows Trill using WDAqua-core1 in the back end for “Give me museums in Lyon.”Source: §6 · p. 13, Figure 4 · p. 13
7. Conclusion and Future Work
The paper presents multilingual, KB-agnostic QA as a portable approach and reports evaluation across multiple benchmarks. It also integrates the approach for end-users and outlines extensions for future work.
7. Conclusion and Future Work
- The evaluation reports qualitative improvements across languages and KBs, quantitative improvements over many existing systems, and scalability to very large KBs such as DBpedia.The approach also queries multiple KBs.Source: §7 · p. 13
- An adapted set of services lets end-users query multiple KBs in multiple languages through one unified interface.Source: §7 · p. 13
- Future work includes parallelizing queries for acceptable response times, adding KBs, recommending RDF dataset designs for QA, and supporting reified data, aggregates, and functions.Source: §7 · p. 13