PaperlayerView exact revision on arXiv

Source-linked AI summary

Middle-Out Decoding

Shikib Mehri, Leonid Sigal

arXiv:1810.11735v1cs.CL

TL;DR

Sequence generation models can lack diversity and external control, and left-to-right decoding makes later outputs depend strongly on earlier ones. The paper proposes middle-out decoding with dual self-attention, reporting improved synthetic de-noising, video-captioning quality comparable to its baseline, and increased diversity and controllability.

  • Problem

    Left-to-right decoding can limit sequence diversity and make it difficult to control whether a particular later word appears.

    Source: §1 · p. 1

  • Method

    The decoder begins from an important middle word or value and expands in both directions, using dual self-attention to model dependencies between outputs.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    The model significantly improves synthetic de-noising, while video-captioning quality is comparable to the sequence-to-sequence baseline.

    Source: §4.1 · p. 6, §4.2 · p. 8

  • Takeaways & Limitations

    The paper reports diverse outputs and effective generation control, including effective use of external input in its video-captioning oracle experiment.

    Source: §4.2 · p. 8, §5 · p. 9

    Video-captioning quality depends on the quality of the middle-word (verb) classifier.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Despite being virtually ubiquitous, sequence-to-sequence models are challenged by their lack of diversity and inability to be externally controlled. In this paper, we speculate that a fundamental shortcoming of sequence generation models is that the decoding is done strictly from left-to-right, meaning that outputs values generated earlier have a profound effect on those generated later. To address this issue, we propose a novel middle-out decoder architecture that begins from an initial middle-word and simultaneously expands the sequence in both directions. To facilitate information flow and maintain consistent decoding, we introduce a dual self-attention mechanism that allows us to model complex dependencies between the outputs. We illustrate the performance of our model on the task of video captioning, as well as a synthetic sequence de-noising task. Our middle-out decoder achieves significant improvements on de-noising and competitive performance in the task of video captioning, while quantifiably improving the caption diversity. Furthermore, we perform a qualitative analysis that demonstrates our ability to effectively control the generation process of our decoder.

1 Introduction

The introduction identifies left-to-right decoding as limiting diversity and external control, then presents a middle-out decoder that starts from an important output and expands in both directions. It also introduces dual self-attention and reports strong results, especially when an important output can be predicted well.

1 Introduction

  • Encoder-decoder models condition sequence generation on an encoded input, commonly represented by a fixed-length hidden vector.The introduction notes that encoded image information can be overwhelmed by the decoder’s learned language prior.Source: §1 · p. 1
  • Left-to-right decoding can limit diversity and makes it difficult to control whether a particular later word appears.The introduction also notes that left-to-right decoding may break natural symmetries in symmetric sequences.Source: §1 · p. 1
  • The middle-out decoder starts from an important word or value and expands in both directions, with dual self-attention supporting consistent generation.The initial output can be automatically predicted or specified directly for controlled generation.Source: §1 · p. 2
  • The paper reports state-of-the-art results and especially improved performance when an important output can be predicted well, such as a video action verb.The introduction identifies sequences with symmetries as another setting where the important element may be predicted well.Source: §1 · p. 2

2 Related Work

Related work situates middle-out decoding among encoder-decoder, self-attentive, and non-traditional sequence-generation approaches. It also contrasts the paper’s controlled decoding with beam-search methods and identifies video captioning as an evaluation setting.

Neural Encoder-Decoder Architectures

  • The proposed decoder focuses on the decoder component of encoder-decoder models used for language and visual captioning.Earlier models used fixed-length representations, while attention allowed decoders to attend to encoded inputs (Bahdanau et al., 2015).Source: §2 · p. 2

Self-attention

Non-traditional Decoders

Controlled Decoding

  • Grid Beam Search forces phrase inclusion during decoding, whereas the paper’s controlled generation is less flexible but outperforms it in a confined, realistic setting (Hokamp and Liu, 2017).Post and Vilar (2018) introduced a faster algorithm with the same effects as Grid Beam Search.Source: §2 · p. 2

Video Captioning

3 Approach

The approach replaces left-to-right generation with a decoder that expands from an important middle word in both directions. Dual self-attention supports dependencies across outputs and helps connect the two decoding directions.

3.1 Baseline Attention Sequence-to-Sequence Model

  • The baseline pairs a bidirectional LSTM encoder with a unidirectional LSTM decoder, using attention over encoder states to produce each output.Its general attention score is used to compute alignment weights over encoder time-steps.Source: §3.1 · p. 3

3.2 Self-Attention Mechanism

  • Dual self-attention combines attention over past embedded outputs and decoder hidden states, supporting non-sequential output dependencies and longer-range gradient paths.The embedded-output attention gives direct information about prior terms; hidden-state attention supports long-term dependencies by counteracting vanishing gradients.Source: §3.2 · p. 4

3.3 Middle-Out Decoder

  • Two LSTM decoders start from the classifier-predicted middle word and alternate left/right generation; dual attention links them to support logically consistent outputs.Attention over outputs and hidden states lets each decoder use information from the other, connecting otherwise disjoint time-steps.Source: §3.3 · pp. 4–5

4 Experiments

The experiments evaluate middle-out decoding on synthetic sequence de-noising and MSVD video captioning. They assess de-noising quality and symmetry, plus caption quality, diversity, and control.

4 Experiments

  • The experiments cover symmetric synthetic-sequence de-noising and MSVD captioning, with captioning assessed for quality, diversity, and control.The MSVD evaluation uses the dataset's standard splits and frame-level features.Source: §4 · p. 5, §4.1 · p. 6, §4.2 · p. 7

4.1 Synthetic De-noising Task

  • Middle-out decoding reduced de-noising MSE by 75% and produced a symmetry MSE 570 times smaller than the sequence-to-sequence baseline.Table 1 reports sequence-generation results for the synthetic de-noising task.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 Video Captioning

  • Dual self-attention applied to the baseline outperformed previous work and significantly improved the middle-out decoder's results.The comparison is reported for video captioning on MSVD.Source: Table 2 · p. 7, §4.2 · p. 8
  • In oracle video captioning, middle-out achieved a METEOR score of 40.9, while ordinary middle-out caption quality was comparable to the sequence-to-sequence baseline.Table 2 covers MSVD captioning results, including model and oracle experiments.Source: Table 2 · p. 7, §4.2 · p. 8
  • Simulated gains in middle-word classifier accuracy corresponded to progressively better captions, and middle-out leveraged provided words better than Grid Beam Search.The classifier-accuracy experiment and the external-input comparison are reported in Tables 3 and 5.Source: §4.2 · p. 8, Table 3 · p. 8, Table 5 · p. 9
  • The middle-out decoder significantly improved caption-set diversity over the baseline in the reported evaluation.Diversity was measured with Self-BLEU and Self-METEOR; Table 4 compares the baseline and middle-out decoder with self-attention.Source: §4.2 · p. 8, Table 4 · p. 8

5 Conclusion

The conclusion presents a middle-out decoder that starts from an important token and expands in both directions, using dual self-attention to support coherent generation. The authors report state-of-the-art results, diverse outputs, and controllability.

5 Conclusion

  • The middle-out decoder starts from an important token and expands the sequence in both directions.Source: §5 · p. 9
  • Dual self-attention models complex dependencies between outputs to support coherent middle-out generation.Source: §5 · p. 9
  • The authors report state-of-the-art results alongside diverse outputs and controllability.Source: §5 · p. 9

6 Supplementary Materials

The supplementary materials provide generated-caption examples from the middle-out and sequence-to-sequence models and expand the qualitative control experiment. They report that middle-word input can shift video attention, while qualifying the sequence-to-sequence model’s apparent tendency to ignore that input.

6 Supplementary Materials

  • The supplementary materials identify Tables 8 and 9 as generated-caption examples and Table 10 as an expansion of the control experiment in Table 6.Source: §6 · p. 13
  • Table 8 gives qualitative examples of captions generated by the middle-out decoder with dual self-attention.Source: Table 8 · p. 13
  • Table 9 presents sequence-to-sequence captions from the paper’s highest-scoring model, which has a METEOR score of 34.4 and outperforms previous work.Source: Table 9 · p. 14
  • Table 10’s control examples show middle-word input can steer the middle-out decoder toward different sections of a video.The sequence-to-sequence model appears to ignore the input in these examples, but its oracle-trained version outperforms the baseline, so it does not ignore the input in every case.Source: Table 10 · p. 15
Loading 1810.11735v1…