Source-linked AI summary
Complex Factoid Question Answering with a Free-Text Knowledge Graph
Chen Zhao, Chenyan Xiong, Xin Qian, Jordan Boyd-Graber
TL;DR
Knowledge-graph QA can miss relations needed for complex factoid questions, while machine-reading models have difficulty synthesizing multiple pieces of evidence. DELFT builds a free-text graph linking question entities to candidate answers and uses a graph neural network to reason over its evidence. Across three QA datasets, it outperforms machine-reading, BERT answer ranking, and a BERT memory-network approach using the same evidence but no graph structure.
Problem
Method
DELFT links question and candidate entities through Wikipedia sentences in a free-text knowledge graph and uses a graph neural network to distinguish useful evidence and combine information across the graph.
Source: §1 · p. 2Results
Across QBLink, QANTA, and TriviaQA, DELFT significantly outperforms machine-reading, BERT answer ranking, and a BERT memory-network approach using the same evidence but no graph structure.
Source: §1 · p. 2Takeaways & Limitations
DELFT’s accuracy improves on more complex questions and when dense evidence is available, whereas machine reading cannot take advantage of additional evidence.
Source: §1 · p. 2A case involving Odysseus and Penelope indicates that more fine-grained reasoning about specific properties may be needed when both candidates share a relation to the same entity.
Source: §8 · p. 10
Abstract
from arXiv · showhide
We introduce DELFT, a factoid question answering system which combines the nuance and depth of knowledge graph question answering approaches with the broader coverage of free-text. DELFT builds a free-text knowledge graph from Wikipedia, with entities as nodes and sentences in which entities co-occur as edges. For each question, DELFT finds the subgraph linking question entity nodes to candidates using text sentences as edges, creating a dense and high coverage semantic graph. A novel graph neural network reasons over the free-text graph-combining evidence on the nodes via information along edge sentences-to select a final answer. Experiments on three question answering datasets show DELFT can answer entity-rich questions better than machine reading based models, bert-based answer ranking and memory networks. DELFT's advantage comes from both the high coverage of its free-text knowledge graph-more than double that of dbpedia relations-and the novel graph neural network which reasons on the rich but noisy free-text evidence.
1 INTRODUCTION
DELFT addresses the limited relation coverage of knowledge graphs and the difficulty of combining evidence from unstructured text for complex factoid questions. It links question and candidate entities with free-text sentences, then uses a graph neural network to select an answer from the diverse, noisy evidence.
1 INTRODUCTION
- Knowledge-graph QA depends on relation coverage, but complex, linguistically rich clues often fall outside closed-form relations, while machine-reading models focus on single-evidence extraction.The paper notes that relation data is costly to obtain and that synthesizing multiple pieces of evidence remains an ongoing research topic.Source: §1 · p. 1
- DELFT connects question entities to candidate answer entities with Wikipedia sentences as evidence edges, rather than relying on predefined relations.The resulting graph supports complex modeling and benefits from the high coverage of free-text evidence.Source: §1 · p. 2
- A graph neural network distinguishes useful evidence from unrelated text and combines information from edge sentences and graph structure to answer questions.DELFT uses this network to reason over informative but noisy relations and multiple sentences associated with each edge.Source: §1 · p. 2
- Across QBLink, QANTA, and TriviaQA, DELFT significantly outperforms machine-reading, BERT answer-ranking, and a BERT memory-network approach using the same evidence but no graph structure.Its accuracy improves on more complex questions and when dense evidence is available, while machine reading cannot take advantage of additional evidence.Source: §1 · p. 2
- Ablations examine node representation, edge representation, and evidence combination, while visualizations and case studies illustrate multi-evidence aggregation.These analyses are presented as evidence about component importance and the model’s ability to combine clues for a more reliable prediction.Source: §1 · p. 2
2 TASK DEFINITION
The paper considers open-domain factoid QA, where natural-language questions ask for entity or attribute facts but may rely on relations absent from knowledge graphs. It grounds questions in a free-text subgraph and reasons over the resulting clues to find an answer entity.
2 TASK DEFINITION
- Questions may involve relationships that closed-form knowledge graphs do not cover, limiting the potential of knowledge-graph QA systems.The paper contrasts benchmarks designed around existing KG relations with real-world questions, where minimal KG coverage precludes broader deployment.Source: §2 · p. 2
- The task is open-domain factoid QA: answer natural-language questions with entities or attributes, without relying on explicit closed-form relations.The evaluated questions are described as human-solvable using knowledge about real-world entities.Source: §2 · p. 2
- For each question, DELFT grounds a subgraph linking question entities to candidate entities through evidence edges, then uses a GNN to distinguish useful clues from noise and aggregate them for prediction.The task description frames answering as extracting multiple clues from different parts of the question.Source: §2 · p. 2
3 FREE-TEXT GRAPH CONSTRUCTION
DELFT builds a free-text knowledge graph from entity co-occurrences in Wikipedia, grounds each question in a candidate-linked subgraph, and prunes that graph to control noise and computation.
3 FREE-TEXT GRAPH CONSTRUCTION
- Rather than extracting explicit relations from text, DELFT uses sentences containing two entities as indirect relations, leaving answer selection to the QA model.The paper argues that OpenIE favors precision over recall and relies on well-defined semantics, while the direct free-text graph avoids this intermediate extraction step.Source: §3 · p. 2
3.1 Graph Construction
- Wikipedia entities form graph nodes, and sentences in which entity pairs co-occur serve as evidence edges; node glosses use the first sentence of each entity’s document.Potential edge sentences are collected from either entity’s page when it mentions the other, or from elsewhere in Wikipedia when both are mentioned.Source: §3.1 · pp. 2–3
3.2 Question Grounding
- For each question, DELFT forms a bipartite subgraph connecting question entities to candidate entities through evidence sentences.Candidates come from entities linked on question-entity pages and entities in top Wikipedia pages retrieved using the question; the two sources capture direct mentions and more thematic matches.Source: §3.2 · p. 3
3.3 Question Graph Pruning
- To reduce graph size and spurious clues, DELFT keeps top-ranked candidate nodes and the five evidence sentences per edge most similar to the question by TF-IDF.Candidate-node relevance is scored with a fine-tuned BERT filter; training keeps the top twenty nodes, while development and test keep the top fifty, and omitting the answer node prevents answering.Source: §3.3 · p. 4
4 FREE-TEXT GRAPH MODELING
DELFT models question answering over a free-text graph, using node and edge information to refine candidate representations. Its GNN scores candidates after passing evidence from question entities through multiple layers.
4 FREE-TEXT GRAPH MODELING
- Connectivity, question-aligned edge evidence, and node relevance are complementary signals for identifying candidate answers.Correct candidates tend to connect to multiple question entities; DELFT also prioritizes question-similar evidence edges and uses entity glosses as node features.Source: §4 · pp. 4–5
4.1 Initial Representations
- Question and entity-gloss text is encoded with an RNN and self-attention to form initial representations.Glosses help identify entity type and whether question entities contain useful clues.Source: §4 · p. 5, §4.1 · p. 5
- Each evidence sentence is encoded and made question-aware, then sentence representations are averaged into an edge representation.The question representation is used in inter-attention when incorporating question information into each edge sentence.Source: §4.1 · p. 5
4.2 Graph Update
- Across layers, candidate representations accumulate information from question nodes, edges, and their previous representations.Stacked updates refine candidate representations before they are used to calculate answer scores.Source: §4.2 · pp. 5–6
- At each graph-update layer, DELFT scores edges by question similarity and passes weighted evidence from question entities toward candidate entities.When multiple sentences link entities, their updated representations are averaged for the edge; candidate updates combine prior candidate state, question representation, and passed information.Source: §4.2 · pp. 5–6
4.3 Answer Scoring
- An MLP scores each candidate entity node from its final-layer representation, and inference selects the candidate with the highest score.Training uses binary cross-entropy over candidate entity nodes.Source: §4.3 · p. 6
5 EXPERIMENTS
The experiments evaluate DELFT on three expert-authored question-answering datasets, with TriviaQA filtered to the paper’s factoid setting. The section also specifies comparison methods and implementation settings.
5 EXPERIMENTS
- The evaluation uses QBLink, Qanta, and TriviaQA; all are entity-rich, while Qanta especially tests relationships between answers and question entities.TriviaQA is relatively simpler and has fewer entities per question; the authors report lower DELFT accuracy on it.Source: Table 1 · p. 6, §5 · p. 6
- For Triviaqa, they filter out questions without a Wikipedia-title answer, keep 70% of questions, and further exclude the 11% in which TagMe finds no entities.Source: §5 · p. 6
5.1 Question Answering Methods
- The comparison includes text-based QA, machine-reading systems, BERT candidate rankers, and a BERT memory network using DELFT’s evidence without graph structure.The BERT memory network concatenates the evidence sentences into a memory cell; DELFT is evaluated with GloVe and BERT embeddings.Source: §5.1 · pp. 6–7
5.2 Implementation
- DELFT keeps the top 20 candidate nodes for training, 50 for testing, and up to five sentences per edge.The implementation uses PyTorch and DGL; BERT output supplies contextualized embeddings for DELFT-BERT.Source: §5.2 · p. 7
- The DrQA setup retrieves 10 documents and then 10 paragraphs, while DocQA uses its pretrained TriviaQA-unfiltered model on the evaluation subset.DrQA inference restricts candidate spans to entities tagged by TagMe.Source: §5.2 · p. 7
6 EVALUATION RESULTS
DELFT’s free-text graph provides broad, dense evidence coverage, and DELFT outperforms the reported baselines on answer accuracy. Ablations and examples examine how graph construction and evidence weighting contribute to its predictions.
6.1 Graph Coverage
- More than 80% of questions remain answerable after filtering to 50 candidates, versus 38% reachable within two DBpedia hops in 50 sampled QBLink questions.Before filtering, each question connects to an average of 1,500+ candidate nodes, and 90% can be answered by connected nodes.Source: §6.1 · p. 7, Table 3 · p. 8
- Correct answers connect to more evidence edges and sentences than incorrect candidates; QBLink and Qanta average five and twelve edges, respectively.TriviaQA questions have two entities on average, with more than one connected to the correct answer; each edge has eight to fifteen evidence sentences.Source: §6.1 · p. 7
6.2 Answer Accuracy
- DELFT outperforms all baselines on the full datasets and entity-count subsets, including machine-reading, BERT ranking, and memory-network comparisons.On TriviaQA, the comparison is on an entity-focused subset; DELFT is close to BERT-MemNN where candidates have fewer than two edges.Source: §6.2 · p. 7, Table 4 · p. 8
6.3 Ablation Study
- Accuracy gains from adding sentences per edge diminish at six; fewer than three reduces accuracy, so DELFT uses five sentences per edge.Using all automatically linked entities slightly hurts accuracy, while thresholds above 0.2 filter too aggressively; the selected threshold is 0.1.Source: §6.3 · p. 7, Figure 3 · p. 8
- On QBLink, node glosses and edge evidence each contribute about 10% accuracy, edge-importance scoring about 3%, and self-attention about 2.2%.The ablations use DELFT-Glove and remove one component while holding other settings constant.Source: §6.3 · pp. 7–8, Table 5 · p. 9
6.4 Graph Visualization
- In the Reagan example, the GNN gives the distinctive Bedtime for Bonzo edge weight 0.96, outweighing shared governor evidence scored 0.06 and 0.04.Aggregating the edges leads DELFT’s GNN to select Ronald Reagan.Source: §6.4 · p. 9, Figure 4 · p. 9
6.5 Case Study
- The case studies show DELFT succeeding with direct or aggregated evidence, and failing when questions have too few informative entities or candidates share overlapping evidence.Source: §6.5 · p. 9
7 RELATED WORK: KNOWLEDGE REPRESENTATION FOR QA
The related work contrasts question answering over structured knowledge graphs with reading-based systems, then situates DELFT among graph-network approaches. It emphasizes the constraints of relation coverage and the challenge of combining evidence across text.
7.1 Knowledge Graph Question Answering
- Knowledge-graph QA methods parse questions into logical forms or align them to graph structures, so their effectiveness depends on available relations.Adding relations takes time and money and may require human effort or structured-resource scraping,,.Source: §7.1 · p. 9
- QEST builds a noisy quasi-graph from extracted entity names and relation phrases, whereas DELFT uses free-text sentences as indirect relations to favor evidence coverage.The passage contrasts QEST’s precision orientation and need for regular, clean text with DELFT’s recall orientation.Source: §7.1 · p. 9
7.2 Question Answering over Text
- Text-reading systems retrieve passages and extract answer spans, but the cited DRQA and ORCA settings mainly require single evidence; DELFT seeks linked edge evidence and nodes.The passage presents synthesizing information across multiple passages as an ideal role for DELFT.Source: §7.2 · pp. 9–10
- Multi-hop QA tests whether systems can assemble information; HotpotQA answers are text spans requiring one or two hops, while span-based DELFT is a future direction.Source: §7.2 · p. 10
7.3 Graph Networks for qa
- Unlike prior graph-QA methods with co-reference or co-occurrence edges, DELFT uses free text as graph edges and represents and aggregates them with a GNN.The cited approaches include Entity-GCN [13], DFGN, HDE [52], and CogQA [15].Source: §7.3 · p. 10
8 THE VIEW BEYOND DELFT
DELFT combines knowledge-graph-style reasoning with high-coverage free-text evidence to answer diverse factoid questions. The paper points to broader applications and finer-grained reasoning as directions for further work.
8 THE VIEW BEYOND DELFT
- DELFT connects entities with free-text edges and uses a graph neural network to select answers from evidence in the text and graph structure.Questions are grounded in related subgraphs connecting entities through free-text edges.Source: §8 · p. 10
- Whether combining natural language and knowledge-rich graphs helps dialog, language modeling, or ad hoc search remains for future work to explore.Source: §8 · p. 10
- More fine-grained reasoning could distinguish candidates that share a relation but differ in a relevant property, as with Odysseus and Penelope.Both have Telemachus as a son, but only Odysseus made a long journey.Source: §8 · p. 10