Source-linked AI summary
KILT: a Benchmark for Knowledge Intensive Language Tasks
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, Sebastian Riedel
TL;DR
Knowledge-intensive NLP tasks need access to large external sources, but differing datasets and infrastructure make general modeling difficult. KILT standardizes eleven datasets across five tasks on one Wikipedia snapshot, evaluates outputs and provenance, and finds promising results from dense retrieval with sequence-to-sequence generation, while provenance remains generally low.
Problem
Knowledge-intensive tasks lack shared infrastructure because datasets use different formats, assumptions, evaluations, and knowledge sources, complicating general comparison and reusable indexing.
Method
KILT builds a benchmark and library that align eleven datasets across five tasks to a common interface and unified Wikipedia snapshot, with provenance for each instance.
Results
A general solution combining dense retrieval and sequence-to-sequence generation shows promising results across tasks, while current models generally provide low-quality provenance.
Takeaways & Limitations
KILT supports cross-task evaluation of general and explainable knowledge-intensive models using shared knowledge representations and provenance-aware assessment.
Takeaways & Limitations
KILT assumes an in-KB setting in which evidence for every instance is present in the knowledge source, leaving unanswerable instances for future work.
Abstract
from arXiv · showhide
Challenging problems such as open-domain question answering, fact checking, slot filling and entity linking require access to large, external knowledge sources. While some models do well on individual tasks, developing general models is difficult as each task might require computationally expensive indexing of custom knowledge sources, in addition to dedicated infrastructure. To catalyze research on models that condition on specific information in large textual resources, we present a benchmark for knowledge-intensive language tasks (KILT). All tasks in KILT are grounded in the same snapshot of Wikipedia, reducing engineering turnaround through the re-use of components, as well as accelerating research into task-agnostic memory architectures. We test both task-specific and general baselines, evaluating downstream performance in addition to the ability of the models to provide provenance. We find that a shared dense vector index coupled with a seq2seq model is a strong baseline, outperforming more tailor-made approaches for fact checking, open-domain question answering and dialogue, and yielding competitive results on entity linking and slot filling, by generating disambiguated text. KILT data and code are available at https://github.com/facebookresearch/KILT.
1 Introduction
KILT addresses the difficulty of developing and comparing knowledge-intensive NLP systems by aligning diverse tasks to one interface and one Wikipedia snapshot. It evaluates general and task-specific approaches for both task performance and provenance.
- Knowledge-intensive tasks require access to large information collections while producing fine-grained outputs such as answers, evidence, or dialogue.
- Different datasets use distinct formats, preprocessing assumptions, evaluations, and knowledge sources, making comparisons and reusable knowledge representations difficult.Changing sources can also require re-indexing documents or retraining models tied to a particular Wikipedia snapshot.
- KILT unifies eleven datasets across five tasks through a model-agnostic interface and a single Wikipedia snapshot.The benchmark covers varied inputs and outputs, including claims, text chunks, structured queries, questions, conversations, and discrete, extractive, or abstractive responses.
- KILT attaches provenance spans to instances so systems can be evaluated on both output accuracy and supporting evidence.An annotation campaign increases provenance coverage, and provenance-aware variants award points only when systems identify supporting Wikipedia pages.
- A neural retriever paired with a pretrained sequence-to-sequence model outperforms most task-specific solutions when trained end-to-end.The benchmark also provides retrieval systems, text-reading models, and an open-source library for knowledge-intensive NLP research.
- Baseline performance on provenance-aware metrics is poor, indicating that current systems still struggle to explain their predictions.
2 Knowledge Source
KILT constructs a unified knowledge source from a fixed Wikipedia snapshot and maps datasets to it through page matching and provenance alignment. The procedure filters evaluation examples when reliable mappings cannot be established.
- The KILT knowledge source uses the 2019/08/01 Wikipedia snapshot containing 5.9M articles.The fixed source must contain the information needed by all included task examples.
- Mapping Datasets to a Fixed Snapshot: Dataset pages are matched to the snapshot, using Wikipedia URL redirections when page titles have changed.
- Mapping Datasets to a Fixed Snapshot: For each matched page, KILT scans the whole page and selects the span with the highest BLEU score against the original provenance span.If BLEU scores tie, the shortest span is returned.
- Mapping Datasets to a Fixed Snapshot: KILT replaces original provenance with the corresponding span from its knowledge source and uses a 0.5 threshold to retain high-quality evaluation mappings.This process discards 18% of test and development data on average, excluding entity linking, while retaining all training pairs.
3 Tasks
KILT covers five knowledge-intensive tasks using a common interface and Wikipedia-based provenance. The tasks vary in inputs, outputs, and knowledge-seeking requirements.
- KILT includes fact checking, open-domain question answering, slot filling, entity linking, and dialogue.
- The benchmark represents each dataset with an input, valid output list, and complementary provenance spans.
- Entity Linking: Entity linking maps a tagged entity mention to a unique Wikipedia page, with the page serving as provenance.
- Slot Filling: Slot filling takes a subject entity and relation as input and outputs equally valid object entities with supporting provenance.
- Open Domain Question Answering: Open-domain question answering requires answering questions without a predefined evidence location, using an entire knowledge source.
- Dialogue: Dialogue uses conversation history to generate the next utterance, with Wizard of Wikipedia grounding one speaker’s utterance in a Wikipedia sentence.
4 Provenance Annotation Campaign
KILT adds provenance annotations through Mechanical Turk campaigns for NQ and ELI5, using passage judgments and salient-span highlighting.
- Annotators judged whether selected passages contained enough evidence and highlighted a salient span for each passage.
- For NQ, gold passages were included to control annotation quality and filter questions unanswerable from the KILT snapshot.
- A passage became provenance when at least two of three annotators judged it sufficient evidence.
- 1507 ELI5 development instances and 600 test instances received provenance, while NQ filtering removed 8% of data on average.
5 Evaluation Metrics
KILT evaluates downstream outputs, evidence retrieval, and their combination through task-specific metrics and provenance-aware KILT scores.
- KILT reports downstream performance, retrieval quality, and a combined measure of output accuracy with supporting evidence.
- Downstream performance: Accuracy, EM, ROUGE-L, and F1 are selected according to whether tasks produce discrete, extractive, short abstractive, or long abstractive outputs.
- Retrieval: Page-level retrieval uses R-precision and Recall@k to measure whether systems retrieve complete provenance page sets.
- Retrieval: For FEVER and HotpotQA, evaluation accounts for provenance sets requiring multiple pages when determining retrieval rank.
- KILT scores: KILT scores award downstream metric points only when R-precision equals 1 for at least one complete provenance set.
6 Baselines
KILT compares retrieval, task-specific, and general baselines across approaches that differ in knowledge representation, prediction style, and task scope.
- The baselines span retrieval, task-specific, and general systems for conditioning outputs on retrieved knowledge.
- The comparison includes explicit or implicit knowledge representations and discriminative, extractive, or generative outputs.
7 Results
Results compare task-specific and general systems on downstream performance, retrieval, and joint output-provenance scores. Explicit retrieval generally improves performance, while retrieval preferences vary by task and provenance scores remain low.
- Explicit knowledge access outperforms implicit pretrained sequence-to-sequence knowledge on almost all datasets.
- End-to-end training of retriever and reader components, as in RAG, produces stronger performance than separate components.
- Entity Linking systems perform well without explicit knowledge access by preserving, altering, or disambiguating entity mention text.
- Retrieval has no clear overall winner: entity-based retrieval benefits Entity Linking and Slot Filling, whereas DPR works better for NQ, FEV, and ELI5.
- Jointly training Multi-task DPR on all KILT data improves performance on every dataset over DPR trained only on NQ and TQA.
- KILT scores jointly evaluate outputs and provenance, but their generally low absolute values leave substantial room for improvement.
8 Discussion
The discussion highlights that custom resources and heuristics can simplify slot filling, while provenance evaluation faces attribution and exhaustiveness challenges.
- Slot filling can be simplified through Wikipedia title lookups or structured resources such as Wikidata.
- KILT uses slot filling to test whether a general model can extract entity attributes from a large textual collection despite such shortcuts.
- Provenance may come from another system and may omit alternative supporting Wikipedia pages, so annotation is used to mitigate these issues.
9 Related Work
KILT builds on benchmarks for knowledge-intensive language tasks while unifying publicly available Wikipedia-based datasets under one knowledge source.
- Prior benchmarks cover natural language understanding, multitask question answering, reading comprehension, question understanding, and dialogue.
- Existing knowledge-intensive resources include TAC-KBP, ARC, TriviaQA-web, Quasar-T, WebQuestions, and ComplexWebQuestions.
- KILT exclusively uses publicly available Wikipedia-based datasets to merge and unify its knowledge source.
10 Conclusion
KILT benchmarks models that condition on a defined Wikipedia snapshot across five task domains. The results support dense retrieval with sequence-to-sequence generation as a promising general approach, while provenance remains weak.
- KILT evaluates models that use specific knowledge from a defined Wikipedia snapshot across five domains.
- Dense retrieval combined with sequence-to-sequence generation yields promising results as a general solution, but substantial improvement remains possible.
- Current models generally provide low-quality provenance for their predictions.
A Appendix
The appendix describes dataset mappings, retrieval systems, baseline models, and checks that KILT's unified Wikipedia setting preserves comparable performance.
- Dataset Mapping Details: KILT maps multiple datasets to a common Wikipedia representation while preserving dataset-specific provenance and answer conventions.The mappings address FEVER, Zero Shot RE, T-REx, Natural Questions, TriviaQA, ELI5, and Wizard of Wikipedia.
- Performance Impact Of The Mapping Strategy: 42.58 EM and 50.43 F1-score are reported for DPR on the mapped Natural Questions development set.The reported development performance is described as on par with prior open-domain results.
- Performance Impact Of The Mapping Strategy: 80.67 dev and 78.93 test accuracy are reported for BART on FEVER under the KILT mapping, including adversarial FEVER 2.0 examples.The authors describe these results as in line with prior work.
- Retrieval Baselines: Retrieval baselines span TF-IDF, dense passage retrieval, and entity-linking-based page ranking.DrQA uses bigram hashing and TF-IDF; DPR uses BERT-based dense passage retrieval; BLINK and flair rank pages using entities.
- Retrieval Baselines: The multi-task DPR jointly trains one dense retriever across eight KILT datasets using uniform resampling to balance tasks.Each task contributes 150k samples per training epoch, with batches drawn from one dataset at a time in round-robin order.
- Task-specific Baselines: Task-specific baselines remain relevant because some tasks may benefit from architectures designed specifically for them.The benchmark therefore evaluates both general-purpose and dedicated approaches.