Source-linked AI summary
Learning a Natural Language Interface with Neural Programmer
Arvind Neelakantan, Quoc V. Le, Martin Abadi, Andrew McCallum, Dario Amodei
TL;DR
Database questions require language understanding and reasoning over tables, while semantic parsers have relied on domain-specific grammars or pruning. This paper adapts Neural Programmer to learn programs from weak answer supervision, achieving 37.7% accuracy with an ensemble of 15 models versus 37.1% for a traditional semantic parser on WikiTableQuestions.
Problem
Answering database questions requires language understanding and discrete reasoning, and semantic parsing maps questions to executable programs.
Source: §1 · p. 1Method
The paper adapts Neural Programmer and its objective to learn table-question programs end-to-end from final-answer supervision, including ambiguous answer-generation decisions.
Source: §1 · p. 3Results
37.7% accuracy from an ensemble of 15 models compares with 37.1% for a traditional semantic parser; a single model reaches 34.2% using 10,000 weakly supervised examples.
Source: §1 · p. 3Takeaways & Limitations
The work presents a weakly supervised, end-to-end neural model that induces programs on a real-world dataset with performance comparable to a state-of-the-art traditional semantic parser.
Source: §5 · p. 10Generating natural-language responses with an RNN decoder is outside the paper's focus.
Source: §2.3 · p. 4
Abstract
from arXiv · showhide
Learning a natural language interface for database tables is a challenging task that involves deep language understanding and multi-step reasoning. The task is often approached by mapping natural language queries to logical forms or programs that provide the desired response when executed on the database. To our knowledge, this paper presents the first weakly supervised, end-to-end neural network model to induce such programs on a real-world dataset. We enhance the objective function of Neural Programmer, a neural network with built-in discrete operations, and apply it on WikiTableQuestions, a natural language question-answering dataset. The model is trained end-to-end with weak supervision of question-answer pairs, and does not require domain-specific grammars, rules, or annotations that are key elements in previous approaches to program induction. The main experimental result in this paper is that a single Neural Programmer model achieves 34.2% accuracy using only 10,000 examples with weak supervision. An ensemble of 15 models, with a trivial combination technique, achieves 37.7% accuracy, which is competitive to the current state-of-the-art accuracy of 37.1% obtained by a traditional natural language semantic parser.
1 BACKGROUND AND INTRODUCTION
This work develops a Neural Programmer-based semantic parser for database questions, addressing the challenge of learning programs from weak answer supervision and ambiguities in real-world tables. On WikiTableQuestions, an ensemble of 15 models achieves 37.7% accuracy, compared with 37.1% for a traditional semantic parser.
1 BACKGROUND AND INTRODUCTION
- Semantic parsing maps natural language to symbolic representations, such as executable logical forms or programs, but requires language understanding and discrete reasoning.Database interaction can require programming skills and schema knowledge; counting and row selection are examples of discrete operations.Source: §1 · p. 1
- Earlier semantic parsers moved from question-program annotations to question-answer supervision, but still relied on domain-specific grammars or pruning strategies.Related neural approaches often required complete programs or searched layouts supplied by a syntactic parse; the latter was evaluated on a simpler dataset.Source: §1 · pp. 1–2
- Neural Programmer selects a table operation and column at each timestep, using a row selector to transfer information and answer variables to store outputs.Soft selection during training enables end-to-end backpropagation from final-answer supervision while the program remains latent.Source: §1 · p. 2, Figure 1 · p. 2
- The approach adapts Neural Programmer for semantic parsing and adjusts its objective to make it more broadly applicable.It integrates neural-network operations with table knowledge and discrete operations on tables and entries.Source: §1 · p. 3
- Backpropagation through answer-generation decisions addresses ambiguities when answers could be generated or selected from repeated table entries.Real-world tables do not mark answer positions, unlike the synthetic dataset used in earlier Neural Programmer work.Source: §1 · p. 3
- 37.7% accuracy from an ensemble of 15 Neural Programmer models compares with 37.1% for a traditional semantic parser; a single model achieves 34.2% using 10,000 weakly supervised examples.WikiTableQuestions test tables are unseen during training, requiring adaptation to unseen column names.Source: §1 · p. 3
2 NEURAL PROGRAMMER
Neural Programmer combines question and table representations with discrete operations to induce programs over tables. Its training objective handles answer-only supervision, including ambiguity over whether an answer comes from a table entry or a scalar operation.
2 NEURAL PROGRAMMER
- The model combines a question LSTM, learned discrete operations, a selector over operations and columns, and a history RNN that tracks earlier selections.It runs for a fixed number of timesteps and learns its parameters by backpropagation from the final answer.Source: §2 · pp. 3–4
2.1 OPERATIONS
- Fifteen manually defined operations act mainly on selected rows; they include counting, entry selection, comparisons, row navigation, and resetting the row selector.The operations are defined for soft selection so they can be trained with backpropagation.Source: §2.1 · p. 4
2.2 OUTPUT AND ROW SELECTOR
- The row selector carries selected rows across timesteps, while lookup answers store selected table entries and scalar answers store numeric answers not provided in the table.The final lookup answer assigns probabilities to table elements as members of the predicted answer.Source: §2.2 · p. 4
2.3 TRAINING OBJECTIVE
- For list answers, the lookup loss uses the highest-probability matching table entry when an answer occurs in multiple positions.The output mask marks which table cells belong to the answer.Source: §2.3 · p. 5
- When a numeric answer also appears in the table, the objective takes a soft minimum of scalar and lookup losses; otherwise it uses the loss matching the answer type.The lookup loss is multiplied by 50.0 to account for the different scales of the two losses.Source: §2.3 · pp. 4–5
3 EXPERIMENTS
The experiments evaluate Neural Programmer on WikiTableQuestions against non-neural baselines and neural sequence models.
3 EXPERIMENTS
- The experiments compare Neural Programmer with non-neural baselines, including the semantic parser of Pasupat & Liang (2015).They also report results for a sequence-to-sequence model and a modified pointer network.Source: §3 · p. 5
- The model was implemented in TensorFlow and took approximately a day to train on one Tesla K80 GPU.Double precision was used because gradients became undefined in single precision.Source: §3 · p. 5
- The authors made their code available in the TensorFlow models repository.Source: §3 · p. 5
3.1 DATA
The experiments use the WikiTableQuestions train, development, and test split, with test tables unseen during training. Training excludes some answer types and tables with at least 100 rows.
3.1 DATA
- The split contains 11,321 training, 2,831 development, and 4,344 test examples, using Pasupat & Liang (2015) tokenization and number and date preprocessing.Source: §3.1 · p. 5
- Table 1 compares Neural Programmer with Pasupat & Liang (2015) baselines and describes the 15-model ensemble as competitive with the state-of-the-art semantic parser.Source: Table 1 · p. 6
- Test tables are unseen during training, requiring adaptation to unseen column names; training uses only tables with fewer than 100 rows.All test examples are considered, while questions with answers that are neither numbers nor phrases selected from the table are ignored during training but penalized in evaluation.Source: §3.1 · p. 6
3.2 TRAINING DETAILS
Training uses four timesteps, 256-dimensional representations, Adam, and mini-batches of 20. The authors use several regularization measures and tune selected hyperparameters on development data.
3.2 TRAINING DETAILS
- The model uses four timesteps, 256-dimensional word, operation, and hidden vectors, and Adam with mini-batches of 20.Adam's epsilon is set to 1e-6, with other parameters at default values.Source: §3.2 · p. 6
- Regularization includes gradient clipping, early stopping, word replacement and word dropout, weight decay, and dropout on recurrent and other model representations.The reported keep probabilities are 0.8 for input and output vectors and selected representations, and 0.9 for recurrent connections and word dropout.Source: §3.2 · p. 6
- Dropout rates, regularization strength, and Adam's epsilon are tuned by grid search on development data.Source: §3.2 · p. 6
3.3 RESULTS
Neural Programmer achieves competitive performance on WikiTableQuestions with weak supervision from a small dataset. The section also reports baseline, ablation, and training-variant results.
3.3 RESULTS
- Dropout, weight decay, and a matched-entry feature for column selection significantly affect model performance.The matched-entry feature indicates whether a table entry in a column matches a question phrase.Source: Table 2 · p. 7
- Neural Programmer achieves competitive performance on a small dataset of 10,000 weakly supervised examples.The reported result is compared with baselines in Table 1.Source: §3.3 · pp. 6–7
- Pre-trained word vectors and language-model pre-training do not improve results, while REINFORCE training with hard selection fails to learn.The authors suggest that searching millions of symbolic programs makes reward-producing programs unlikely and parameter updates infrequent.Source: §3.3 · p. 7
3.3.1 NEURAL NETWORK BASELINES
- Sequence-to-sequence and modified pointer-network baselines reach only 8.9% and 4.0% development accuracy, respectively.The sequence-to-sequence model is evaluated on examples with input length below 400; the pointer network is trained only on lookup questions with tables under 100 rows.Source: §3.3.1 · p. 7
3.4 ANALYSIS
The analysis examines design choices, induced programs, operation use, and error sources. It finds evidence of adaptation to unseen column names alongside inefficient programs, overfitting, and unanswerable questions.
3.4.1 MODEL ABLATION
- A matched-entry feature for column selection significantly improves performance, while anonymizing question phrases has only a small positive effect.Regularization has a much larger effect than phrase anonymization.Source: §3.4.1 · p. 7
3.4.3 CONTRIBUTION OF DIFFERENT OPERATIONS
- Among correctly answered development examples, 30.7% of programs yield scalar answers; print and select are used 69.3% and 66.7% of the time, respectively.The remaining programs yield table lookups.Source: Table 4 · p. 9, §3.4.3 · p. 9
3.4.2 INDUCED PROGRAMS
- Induced programs can adapt to unseen column names, but some are inefficient or correct only when a select operation matches an entry.Some examples yield the correct development answer without producing programs that work for all input tables.Source: Table 3 · p. 8, §3.4.2 · p. 9
3.4.4 ERROR ANALYSIS
- The model overfits despite strong regularization, and 21% of a random 200-question sample was unanswerable because of issues including annotation errors and advanced table normalization.The training-set accuracy is 53%, about 20% higher than development and test accuracy; the authors suggest more training data could improve performance.Source: §3.4.4 · p. 9
4 OTHER RELATED WORK
Related work includes semi-supervised semantic parsing and neural program-induction models, alongside neural approaches to question answering and reading comprehension.
4 OTHER RELATED WORK
- Kocisky et al. (2016) develop a semi-supervised semantic parser trained with question-program pairs.Source: §4 · p. 10
- Liang et al. (2016) propose a Neural Programmer-like neural symbolic machine trained with REINFORCE, using two operations and three timesteps.The paper describes its induced programs as much simpler than Neural Programmer's.Source: §4 · p. 10
- Neural networks have also been applied to question-answering datasets requiring little arithmetic reasoning, and Wang & Jiang (2016) report state-of-the-art reading-comprehension results.Source: §4 · p. 10
5 CONCLUSION
The paper enhances Neural Programmer for weaker supervision and broader applicability, using soft selection to explore program space by backpropagation with superior sample complexity.
5 CONCLUSION
- Soft selection lets Neural Programmer explore the program space by backpropagation with superior sample complexity, supporting weakly supervised program induction on a real-world dataset.The paper describes this as its first instance of a weakly supervised, end-to-end neural network inducing programs on a real-world dataset.Source: §5 · p. 10
APPENDIX
The appendix lists the model's operations and definitions and describes how the row selector is computed.
APPENDIX
- Table 5 lists the operations provided to the model and their definitions.Source: APPENDIX · p. 13
- The row selector is computed from the outputs of the operations other than count and print.Count and print outputs are used to calculate scalar answer and lookup answer, respectively.Source: ROW SELECTOR · p. 13
- The operation and column selectors assign probabilities to operations and columns at each timestep.Source: ROW SELECTOR · p. 13