PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Descriptions from Structured Data Using a Bifocal Attention Mechanism and Gated Orthogonalization

Preksha Nema, Shreyas Shetty, Parag Jain, Anirban Laha, Karthik Sankaranarayanan, Mitesh M. Khapra

arXiv:1804.07789v1cs.CLcs.AIcs.LG

TL;DR

Generating descriptions from structured facts is not well served by generic seq2seq models that do not exploit task-specific structure. The paper combines attention over fields and their values with gated orthogonalization, reporting a 21% relative improvement over an existing state-of-the-art model on WIKIBIO and state-of-the-art results on French and German datasets.

  • Problem

    Generic seq2seq models treat infoboxes as field-value sequences but do not exploit task-specific characteristics, while varying fields and ordering make simple templates difficult.

    Source: §1 · pp. 1–2

  • Method

    The model fuses macro-level attention over fields with micro-level attention over values and uses gated orthogonalization to model staying on, then not returning to, a field.

    Source: §1 · p. 2

  • Results

    21% relative improvement over an existing state of the art model on a large scale WIKIBIO dataset, with state-of-the-art results on French and German datasets.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Fine-tuning an out-of-domain model with small amounts of in-domain data can improve performance on the target domain.

    Source: §6 · p. 9

Abstract

from arXiv · show

In this work, we focus on the task of generating natural language descriptions from a structured table of facts containing fields (such as nationality, occupation, etc) and values (such as Indian, actor, director, etc). One simple choice is to treat the table as a sequence of fields and values and then use a standard seq2seq model for this task. However, such a model is too generic and does not exploit task-specific characteristics. For example, while generating descriptions from a table, a human would attend to information at two levels: (i) the fields (macro level) and (ii) the values within the field (micro level). Further, a human would continue attending to a field for a few timesteps till all the information from that field has been rendered and then never return back to this field (because there is nothing left to say about it). To capture this behavior we use (i) a fused bifocal attention mechanism which exploits and combines this micro and macro level information and (ii) a gated orthogonalization mechanism which tries to ensure that a field is remembered for a few time steps and then forgotten. We experiment with a recently released dataset which contains fact tables about people and their corresponding one line biographical descriptions in English. In addition, we also introduce two similar datasets for French and German. Our experiments show that the proposed model gives 21% relative improvement over a recently proposed state of the art method and 10% relative improvement over basic seq2seq models. The code and the datasets developed as a part of this work are publicly available.

1 Introduction

The paper targets description generation from structured facts, where generic sequence models miss task-specific structure. It combines attention over fields and their values with mechanisms for retaining and then forgetting fields, and reports improvements on English, French, and German datasets.

1 Introduction

  • Infoboxes pair entity properties with values, while varying fields and field order across diverse biographies make simple rule-based templates difficult.The dataset is described as containing 700K examples across people from backgrounds including sports, politics, and arts.Source: §1 · p. 1
  • Fused bifocal attention combines macro-level attention over fields with micro-level attention over values, letting field attention influence attention to its values.The decoder receives a fused context vector containing field-level and word-level information.Source: §1 · p. 2
  • A forget-or-remember gate models staying on a field for several timesteps, while gated orthogonalization makes later field contexts different from those of a forgotten field.This is intended to model the never-look-back behavior after a field's information has been rendered.Source: §1 · p. 2
  • 21% and 20% relative improvements over current state-of-the-art models, and 10% over basic seq2seq, are reported on WIKIBIO; French and German results also reach state of the art.The WIKIBIO dataset contains around 700K infobox-description pairs.Source: §1 · p. 2

2 Related work

Prior NLG work spans rule-based, statistical, hybrid, and neural approaches, while much structured-data generation research used small datasets. The paper situates its work against larger biography data and neural models for other structured-data tasks.

2 Related work

  • NLG approaches span rule-based, modular statistical, hybrid, and neural models.The passage cites examples across these approaches, including Bahdanau et al. (2014) for neural models.Source: §2 · p. 2
  • Structured-data NLG had largely used small datasets such as WEATHERGOV and ROBOCUP, while Mei et al. (2016) proposed attention-based neural encoder-decoder models for them.The passage contrasts this work with data-hungry neural NLG models trained on large amounts of data.Source: §2 · p. 3
  • Lebret et al. (2016) introduced a larger biography dataset with 700K pairs and a 400K-word vocabulary; this paper adds seq2seq components for structured-data attention and field-decoding behavior.Prior two-level attention work addressed unstructured hierarchies of sentences and words, unlike this paper's fields and values.Source: §2 · p. 3

3 Proposed model

The model extends seq2seq generation from infoboxes with attention to both fields and values, plus gated orthogonalization to model when to stay on or move away from a field.

3 Proposed model

  • The model treats an infobox as field–value pairs and extends seq2seq with bifocal attention and gated orthogonalization for task-specific structure.Source: §3 · p. 3

3.1 Fused Bifocal Attention Mechanism

  • Bifocal attention represents fields and values separately, selecting a field at macro level and values within it at micro level.Source: §3.1 · pp. 3–4
  • It fuses the levels by reweighting each value's micro-attention with the macro-attention of its field.Source: §3.1 · p. 4

3.2 Gated Orthogonalization for Modeling Stay-On and Never Look Back behaviour

  • A remember gate preserves field context across timesteps, while soft orthogonalization reduces overlap with prior field vectors to support moving on.Source: §3.2 · p. 5

4 Experimental setup

The experiments use English WIKIBIO and introduce French and German biography datasets, compare three model families, and tune their hyperparameters.

4.1 Datasets

  • English WIKIBIO contains 728,321 English Wikipedia biographies, with each person’s infobox as input and the article’s first sentence as target.The experiments use the train, validation, and test splits released by Lebret et al. (2016).Source: §4.1 · p. 5
  • French and German datasets contain 170K and 50K examples, respectively, with vocabularies of 297K and 143K; each uses random 80/10/10 train/test/validation splits.They were built from Wikipedia infoboxes paired with the article’s first sentence.Source: §4.1 · p. 5

4.2 Models compared

4.3 Hyperparameter tuning

  • Hyperparameter tuning retains the top 20K words, initializes embeddings with 300-dimensional GloVe, and uses Adam at learning rate 0.0004 with early stopping.GRU state sizes of 128, 256, and 512 were tested; training ran for at most 20 epochs with patience 5.Source: §4.3 · p. 6

5 Results and Discussions

The final model outperforms baselines on English WIKIBIO, with human evaluations, multilingual results, attention visualizations, and out-of-domain experiments providing complementary findings.

5.1 Comparison of different models

  • The final model is 10% (relative) better than the closest baseline (basic seq2seq) and 21% (relative) better than the current state of the art method (Lebret et al., 2016).Source: §5.1 · p. 6

5.2 Human Evaluations

  • Basic seq2seq scored 4.04/3.6 and the final model 4.19/3.9 for average fluency/adequacy; annotators called 423 out of 500 final-model summaries natural.Source: §5.2 · p. 6

5.3 Performance on different languages

5.4 Visualizing Attention Weights

  • In the English illustration, gated orthogonalization keeps attention on the name field while needed, then moves away without returning; differences were more pronounced for French and German.Source: §5.4 · pp. 7–8, Figure 3 · p. 8

5.5 Out of domain results

  • Testing unseen domains significantly reduced performance; fine-tuning with small target-domain samples improved it, while training from scratch on 10K examples yielded BLEU scores of 16.2 for arts and 28.4 for sports.Source: Table 6 · p. 8, §5.5 · pp. 8–9

6 Conclusion

The conclusion presents fused bifocal attention and gated orthogonalization as task-specific decoding components, reports results across three languages, and proposes a direction for multilingual learning.

  • Fused bifocal attention and gated orthogonalization address two-level attention and stay-on/never-look-back behavior during decoding.Source: §6 · p. 9
  • The final model outperforms an existing state-of-the-art model on large-scale WIKIBIO by 21% and gives state-of-the-art results on new French and German datasets.Source: §6 · p. 9
  • Fine-tuning an out-of-domain model with small amounts of in-domain data improves performance on the target domain.Source: §6 · p. 9
  • Future work proposes language-agnostic Wikidata concept IDs to share input vocabulary for joint multilingual learning.Source: §6 · p. 9
Loading 1804.07789v1…