WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia

1 Introduction

• WIKIREADING predicts textual values from Wikidata using the corresponding Wikipedia article text, across 18.87M instances and 867 sub-tasks.The task combines classification and extraction and has a rich output space of millions of answers.Source: §1 · p. 1
• Table 1 illustrates the task’s variety: some answers can be extracted from the article, while others require classification or another form of inference.The input consists of the document and property, and the target is the answer.Source: §1 · p. 1, Table 1 · p. 2
• 71.8% was the highest score, achieved by a word-level sequence-to-sequence model with special handling for out-of-vocabulary words.The comparison covers a suite of recent models evaluated on common grounds.Source: §1 · pp. 1–2

2.1 Wikidata

• Wikidata represents information about items as (property, value) statements and contains over 80M statements across more than 800 properties.Items may be linked to Wikipedia articles.Source: §2.1 · p. 2

2.2 Dataset

• WIKIREADING consolidates statements into (item, property, answer) triples, then replaces linked items with Wikipedia article text to form dataset instances.The strings are human-readable; timestamps are converted to human-readable formats, and the data is split 85/10/5 into training, validation, and test sets.Source: §2.2 · p. 2

2.3 Documents

• The dataset covers 4.7M Wikipedia articles, about 80% of English Wikipedia, with mean and median article lengths of 489.2 and 203 words.Articles average 5.31 instances each, with a maximum of 879.Source: §2.3 · p. 2

2.4 Properties

• Properties divide into categorical (<0.7 scaled entropy; 56.7% of instances) and relational (43.3%), across a skewed set of 867 properties.The 20 most frequent properties cover 75% of instances, and 180 properties cover 99%.Source: §2.4 · pp. 2–3

2.5 Answers

• 54.7% of answers are unique, and 30.0% of test-set answers never occur in training, so some answers must be read from the document.The answer distribution has frequent head answers and many rare tail answers; methods focused only on either end cannot achieve maximum performance.Source: §2.5 · p. 3

3 Methods

• The methods draw on neural architectures that can combine a property and document through sequential input or attention over predictive words and phrases.The property can serve as a question, either prepended to the document in sequential models or used to compute attention.Source: §3 · p. 3
• The authors implement a range of recent models and compare them on common grounds because neural architecture research remains ongoing.They group methods by broad approach and note necessary modifications.Source: §3 · p. 3
• The section also introduces novel variations of the described models.The methods are grouped into broad categories by general approach.Source: §3 · p. 3

3.1 Answer Classification

• Each possible answer is a class label, but the answer space is too large to model exhaustively, so classification covers a tractable subset.Source: §3.1 · p. 3

3.1.1 Baseline

• The baseline concatenates separate bag-of-words vectors for the document and property so one article can receive different predictions by property.Source: §3.1.1 · p. 3

3.1.2 Neural Network Methods

• Neural methods form a joint document–property representation for answer prediction, using averaged embeddings, fixed paragraph vectors, recurrent readers, attention, or memory networks.Averaged Embeddings separately averages document and property word embeddings; the Attentive Reader uses the property to attend over document word representations, while the Memory Network attends over sentence memories.Source: §3.1.2 · p. 4
• Word inputs outside the frequent-word vocabulary map to a shared learned out-of-vocabulary embedding.Source: §3.1.2 · p. 4

• Relational properties map entities to values and are less suited to document classification, motivating extraction of a text span related to a subject.Source: §3.2 · p. 5
• The RNN Labeler uses a word-level RNN to estimate at each position whether the current word belongs to an answer.Source: §3.2 · p. 5
• Training labels answer words as 1 and other positions as 0, requires full sequence matches for multi-word answers, and discards training instances whose answers are absent from the document.Source: §3.2 · p. 5
• At test time, consecutive positions above a threshold are chunked into answers, and the top-scoring answer is submitted.The threshold is 0.5; chunk scores use the harmonic mean of the predicted probabilities of their words.Source: §3.2 · p. 5

3.3 Sequence to Sequence

• Seq2seq combines an encoder that maps an input sequence to a vector with a decoder that generates output tokens, supporting both answer classification and extraction.Source: §3.3 · p. 5

3.3.1 Basic seq2seq

• Basic seq2seq adds an RNN decoder to the LSTM Reader and can produce answer sequences from the document vocabulary.Source: §3.3.1 · p. 5

3.3.2 Placeholder seq2seq

• Placeholder seq2seq replaces document OOV words and matching target words with placeholders, extending the output space to answers the RNN Labeler can extract.It uses word-level placeholders without entity recognition.Source: §3.3.2 · pp. 5–6

3.3.3 Basic Character seq2seq

• Basic character seq2seq encodes the property and document character by character, then decodes the answer as a character sequence.Source: §3.3.3 · p. 6, Figure 2 · p. 6

3.3.4 Character seq2seq with Pretraining

• Character seq2seq pretraining first trains a character-level language model on training-set input sequences, then uses its weights to initialize the encoder's first layer and decoder.The paper notes that character-level sequences are much longer, requiring more recurrent steps and longer gradient propagation.Source: §3.3.4 · p. 6

4 Experiments

• Mean F1 averages per-instance set-overlap F1 scores, allowing partial credit for multi-value answers; the evaluation uses the full test set.Because this paper evaluates single-value answers and most dataset answers are single values, Mean F1 is closely related to accuracy; its stated single-value upper bound is 0.963.Source: §4 · p. 6, §4.2 · p. 7

4.1 Training Details

• Training used shared preprocessing and comparable hyperparameters where possible, with document inputs truncated to 300 words except for Character seq2seq, which used 400 characters.The models were optimized with Adam on batches of 128, and 50 randomly sampled learning-rate and gradient-clipping configurations were compared using validation Mean F1.Source: §4.1 · pp. 6–7

4.2 Results and Discussion

• Attentive Reader outperforms Averaged Embeddings on difficult relational properties and properties with fewer training examples, while classifier accuracy declines for less frequent answers.The authors suggest that learning separate softmax weights for each answer may limit generalization across answers.Source: §4.2 · pp. 7–8, Figure 3 · p. 8
• RNN Labeler performs better on relational than categorical properties and approaches its 0.434 Mean F1 upper bound despite being limited to answers appearing verbatim in the document.Its date-property improvement over classifiers demonstrates its ability to identify answers typically present in the text.Source: §4.2 · p. 8
• Seq2seq models balance performance across relational and categorical properties, with results reported for selected properties to illustrate long-tail behavior.They perform especially well when answers contain rare words; the Basic seq2seq often does slightly better for categorical or date properties.Source: §4.2 · pp. 8–9, Table 5 · p. 8
• Character seq2seq models perform comparably to word-level seq2seq despite copying long character strings and accessing a smaller portion of the document.Language-model pretraining improves their performance, especially on relational properties, reduces hyperparameter sensitivity, and accelerates training.Source: §4.2 · p. 9

5 Related Work

• Wu and Weld (2007) pursued structured extraction from Wikipedia infoboxes; Kylin was restricted to four infobox classes and common attributes.The passage contrasts infoboxes with Wikidata, whose data has fewer schema variations related to page formatting.Source: §5 · p. 9
• Distant supervision assumes that sentences containing both knowledge-base triple arguments express the relation, and Mintz et al. (2009) applied it to Freebase triples in Wikipedia.Later extensions model whether a relation is expressed and jointly reason over larger sets of sentences and relations (Riedel et al., 2010; Surdeanu et al., 2012).Source: §5 · p. 9
• Rocktäschel et al. (2015) reduced the number of distant-supervision examples required by sharing information between relations.Source: §5 · p. 9

6 Conclusion

• End-to-end sequence-to-sequence models were the most promising among the compared classification and extraction models, learning to classify documents and copy arbitrary strings.Source: §6 · p. 9
• Character-level model performance improved substantially after language-model pretraining, suggesting further training optimizations may yield gains.Source: §6 · p. 9
• Document length poses a problem for RNN-based models, while convolutional networks may address it and are easier to parallelize.Source: §6 · p. 9
• The authors state that the models rely on little or no traditional NLP preprocessing and suggest testing generalization to other languages with multilingual WIKIREADING.Source: §6 · p. 9