Source-linked AI summary
Character-Level Question Answering with Attention
David Golub, Xiaodong He
TL;DR
Mapping natural-language questions to structured knowledge-base queries is difficult because of paraphrases, unseen entries, and large candidate spaces. The paper uses a character-level, attention-based encoder-decoder for single-relation question answering and jointly learns question, entity, and predicate representations. It achieves higher SimpleQuestions accuracy with a compact model trained on less data and robust to unseen entities.
Problem
Mapping natural-language questions to structured knowledge-base queries is difficult because questions have paraphrases, unseen entries, and large numbers of entities and predicates.
Method
A character-level, attention-based encoder-decoder jointly learns question, entity, and predicate embeddings to generate single-relation knowledge-base queries.
Results
70.9% SQ accuracy on FB2M and 70.3% on FB5M surpass previous state-of-the-art results of 62.7% and 63.9%, respectively.
Takeaways & Limitations
The model is compact, requires less training data, and generalizes to unseen entities while using only the SimpleQuestions training questions.
Takeaways & Limitations
Among sampled errors, the model frequently confuses predicates with very similar aliases and sometimes predicts the wrong entity.
Abstract
from arXiv · showhide
We show that a character-level encoder-decoder framework can be successfully applied to question answering with a structured knowledge base. We use our model for single-relation question answering and demonstrate the effectiveness of our approach on the SimpleQuestions dataset (Bordes et al., 2015), where we improve state-of-the-art accuracy from 63.9% to 70.9%, without use of ensembles. Importantly, our character-level model has 16x fewer parameters than an equivalent word-level model, can be learned with significantly less data compared to previous work, which relies on data augmentation, and is robust to new entities in testing.
1 Introduction
The paper addresses the difficult task of mapping natural-language questions to structured knowledge-base queries. It proposes a character-level encoder-decoder designed to handle paraphrases, unseen entries, and large knowledge bases while improving accuracy with less data.
- Mapping natural-language questions to corresponding knowledge-base queries remains challenging.
- Three obstacles are paraphrases, unseen knowledge-base entries, and millions of entities with thousands of predicates.
- The model uses character-level representations to improve robustness to unseen entities and predicates.
- The encoder-decoder architecture encodes questions with an LSTM and attention, then decodes candidate entities or predicates using prior knowledge-base aliases.
- 70.9% accuracy on Freebase2M and 70.3% on Freebase5M exceed previous state-of-the-art results of 62.7% and 63.9%, respectively.
- The model uses only SimpleQuestions training questions, covering about 24% of test-set entity-alias words, without the data augmentation used in previous work.
2 Related Work
Prior work frames knowledge-base question answering as semantic parsing and commonly uses word-level embeddings for large knowledge bases. This paper instead formulates answering as decoding a query in a unified, end-to-end encoder-decoder framework.
- Semantic parsing translates questions into structured knowledge-base queries for open-domain question answering.
- Large-scale knowledge-base parsers must generate queries involving millions of entities and thousands of predicates, many unseen during training.
- Word-level embeddings can suffer from out-of-vocabulary problems on unseen words, motivating substantial data augmentation in prior approaches.
- Encoder-decoder networks encode a source sequence into a fixed-length representation and decode it into a target sequence.
- This paper formulates question answering as decoding a knowledge-base query from the question and knowledge-base entries.
- The question and knowledge-base embeddings are learned in one unified system optimized end-to-end.
3 Model
The model encodes questions, entities, and predicates at the character level, then uses an attention-based decoder and semantic relevance scoring to generate a two-element knowledge-base query.
- Model overview: The model maps each question and candidate sets of entities and predicates to likelihood scores for all entity–predicate pairs.Its output query contains exactly the topic entity and predicate.
- Question encoder: A character-level LSTM encoder represents the question as a sequence of character embeddings, preserving spaces, punctuation, and case.The question characters are processed left to right by a two-layer gated-feedback LSTM.
- Entity and predicate encoders: Character-level CNN encoders produce single embedding vectors for entities and predicates from their English aliases.Separate CNNs are used because entity names and predicate strings have different styles, and CNNs improve computational efficiency.
- Decoder and relevance scoring: An attention-based LSTM decoder combines question context with a pairwise semantic relevance function to select the topic entity and predicate.The relevance function compares decoder hidden states with candidate embeddings, using cosine similarity in the current experiments.
- Inference: Candidate generation keeps entities whose aliases occur in the question, retains the top 10 entities per alias, and includes their predicates.The model then decodes the most likely entity–predicate pair from these candidate sets.
- Training: The model maximizes the joint likelihood of the correct entity and predicate using back-propagation, with parameters learned jointly.Training uses the true entity and predicate plus 50 randomly sampled entities and predicates, respectively.
4 Dataset and Experimental Settings
The evaluation uses SimpleQuestions questions paired with Freebase triples and compares two candidate-generation settings that contain 2M or 5M entities.
- Dataset: SimpleQuestions contains 108,442 single-relation questions paired with Freebase topic-entity, predicate, and answer-entity triples.The split contains 75,910 training, 10,845 validation, and 21,687 test questions.
- Dataset: Only 10,843 of 45,335 unique entity-alias words and 886 of 1,034 unique predicates in the test set occur in training.These counts characterize the dataset’s unseen vocabulary and predicate coverage.
- Evaluation settings: FB2M and FB5M use Freebase candidate-generation subsets containing 2M and 5M entities, respectively.The two settings test the model with different candidate-set scales.
- Baselines and training data: MemNN baselines use additional WebQuestions, WikiAnswers paraphrases, and automatically generated KB questions, whereas the proposed model uses only the SimpleQuestions training questions.The additional MemNN data include 3K WebQuestions items, 15M paraphrases, and 11M or 12M generated questions.
- Implementation settings: The proposed models use two 200-unit LSTM encoder layers, a 100-unit decoder hidden layer, and CNN entity and predicate embeddings with 200-unit hidden layers and 100-unit outputs.Training uses RMSProp with a learning rate of 1e−4.
- Input preprocessing: Entity and predicate strings, as well as questions, are padded with three padding symbols plus special start and end symbols before encoding.For example, “Obama” receives start, padding, and end symbols.
5 Results
The experiments show that character-level modeling improves question-answering accuracy, especially for unseen entities, while attention reveals distinct representations for entities and predicates.
- End-to-end Results on SimpleQuestions: 70.9% and 70.3% SQ accuracy were achieved on FB2M and FB5M, outperforming previous state-of-the-art results by 8.2% and 6.4%.SQ accuracy counts questions for which both the topic entity and predicate are correct.
- Character-Level vs. Word-Level Models: The character-level model used 1.2M parameters, compared with 19.9M for the word-level model, whose best SQ accuracy was 53.9%.The word-level model also faced a more severe OOV problem because no data augmentation covered unseen entities.
- Character-Level vs. Word-Level Models: Character-level models predicted entities with 96.6% accuracy, whereas word-level models reached only 45% from the mixed candidate set.Both model types reached around 80% predicate accuracy, indicating that the larger difference concerned entity generalization.
- Depth Ablation Study: A two-layer LSTM increased joint accuracy by over 6%, with most gains attributed to improved predicate predictions.The authors suggest entity accuracy may already have been saturated in this experimental setup.
- Attention Mechanisms: Attention typically peaks at spaces, while entity attention spans longer character regions and predicate attention often becomes sharper after complete words.The examples suggest entities become identifiable from partial names, whereas predicates may require complete phrases or sentence-ending information.
- Attention Mechanisms: The model can attend to disjoint regions and use words such as “university” when predicting both the entity and predicate.This avoids requiring a hard separation between entity and predicate mentions.
6 Error Analysis
Error analysis shows that most sampled failures involved predicates with aliases very similar to the correct predicate, while entity errors were also common.
- 46 of 50 errors involved predicting a predicate with an alias very similar to the true predicate.Examples include “/music/release/track” versus “/music/release/track list”.
- 21 of 50 errors involved predicting the wrong entity.One example confused “Album” with “Still Here” in a question about the album type.
- 18 of 50 errors involved predicting both the wrong entity and the wrong predicate.The analysis gives an example involving “Play” and “/freebase/equivalent topic/equivalent type”.
- The authors suggest more training data, expanded negative samples, and examples distinguishing similar predicates as possible remedies.
7 Conclusion
The paper presents a character-level, attention-based encoder-decoder for single-relation question answering with structured knowledge bases. It improves SimpleQuestions accuracy, uses less data, remains compact and robust to unseen entities, and is intended for future extension to multirelation questions.
- The proposed system is a character-level, attention-based encoder-decoder for question answering with a structured knowledge base.
- It jointly learns question, entity, and predicate embeddings to optimize generation of the correct knowledge-base query.
- The model improves state-of-the-art accuracy on SimpleQuestions while using much less data than previous work.
- Character-level modeling yields a compact model robust to unseen entities and supports higher-level semantic concepts through attention.