PaperlayerView exact revision on arXiv

Source-linked AI summary

It was the training data pruning too!

Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan, Kedar Dhamdhere

arXiv:1803.04579v1cs.LGcs.CL

TL;DR

The paper investigates the source of KDG’s improvement, including the KDG paper’s claim that it stems from model architecture. The paper compares KDG trained with its pruned logical-form candidates against KDG trained on unpruned candidates. Disabling pruning drops KDG accuracy from 43.3% to 36.3%.

  • Problem

    The paper investigates the source of KDG’s improvement, including the KDG paper’s claim that it stems from model architecture.

    Source: §1 · p. 1

  • Method

    The paper compares KDG trained with its pruned logical-form candidates against KDG trained on unpruned candidates.

    Source: §2.2 · p. 2

  • Results

    Disabling pruning drops KDG accuracy from 43.3% to 36.3%.

    Source: §1 · p. 1, §2.2 · p. 2

  • Takeaways & Limitations

    Pruning may also be useful for training other semantic parsers, including scoring models and neural parsers using reinforcement learning.

    Source: §2.3 · p. 2

    The finding does not show that pruning is the only important factor because architectural innovations remain essential to KDG’s improvement.

    Source: §3 · p. 3

Abstract

from arXiv · show

We study the current best model (KDG) for question answering on tabular data evaluated over the WikiTableQuestions dataset. Previous ablation studies performed against this model attributed the model's performance to certain aspects of its architecture. In this paper, we find that the model's performance also crucially depends on a certain pruning of the data used to train the model. Disabling the pruning step drops the accuracy of the model from 43.3% to 36.3%. The large impact on the performance of the KDG model suggests that the pruning may be a useful pre-processing step in training other semantic parsers as well.

1 Introduction

The paper revisits why KDG improves question answering on WikiTableQuestions, finding that training-data pruning contributes alongside architectural factors.

  • 43.3% single-model accuracy makes KDG the strongest reported model, compared with 34.8% for the previous best single model.Source: §1 · p. 1
  • The analysis uses a single model because it is easier to analyze than the 5-model ensemble, which achieves 45.9% accuracy.Source: §1 · p. 1
  • The study tests whether KDG’s improvement comes only from the architectural aspects identified in prior ablations.Source: §1 · p. 1
  • Pruning training data is crucial: without it, KDG model accuracy drops to 36.3%.Source: §1 · p. 1

2 KDG Training Data

KDG generates logical-form supervision from answer-only data, and the paper traces a major performance effect to removing spurious candidates before training.

2 KDG Training Data

  • KDG translates each question-table pair into an executable Lambda-DCS logical form whose execution produces the answer.Source: §2 · p. 1
  • Because WTQ provides only answer labels, KDG uses DPD to enumerate logical forms that yield the correct answer for each question-table-answer triple.Source: §2 · pp. 1–2

2.1 Pruning algorithm

  • Spurious logical forms can match an answer accidentally, such as selecting the first row when the table happens to be sorted.Source: §2.1 · p. 2
  • Pruning uses perturbed fictitious tables and removes a logical form if it fails on any such table; an analysis eliminated 92.1% of spurious forms.Source: §2.1 · p. 2, §2.2 · p. 2

2.2 Importance of pruning for KDG

  • 36.3% accuracy after training on unpruned candidates, down from 43.3%, shows that pruning is necessary for KDG’s reported performance improvement.Source: §2.2 · p. 2

2.3 Directions for further work

  • The pruning procedure could support other parsers by down-scoring spurious forms, restricting reinforcement-learning rewards, or augmenting WTQ with fictitious tables.Source: §2.3 · p. 2

3 Discussion

The discussion places pruning within a broader account of KDG’s gains, while emphasizing that architectural innovations remain essential and attribution requires considering multiple system factors.

  • Prior KDG ablations examined architectural innovations but did not consider pruning of the training logical forms.Source: §3 · pp. 2–3
  • Hyper-parameter tuning might slightly improve the unpruned model, but the authors do not expect it to erase the large performance drop.Source: §3 · p. 2
  • Pruning is an important factor in KDG’s improvement, not the only one; the architectural innovations are also essential.Source: §3 · p. 3
  • Machine-learning performance depends on architecture, training algorithm, preprocessing, and hyper-parameters, so attribution should consider all relevant factors.Source: §3 · p. 3

Related papers

Loading 1803.04579v1…