PaperlayerView exact revision on arXiv

Source-linked AI summary

Describing Multimedia Content using Attention-based Encoder--Decoder Networks

Kyunghyun Cho, Aaron Courville, Yoshua Bengio

arXiv:1507.01053v1cs.NEcs.CLcs.CVcs.LG

TL;DR

The paper examines structured-output problems in which input and output structures are related, especially the challenge of aligning output elements with relevant input content. It reviews attention-based encoder–decoder systems built from gated recurrent and convolutional networks, reporting strong results across translation, captioning, video description, and speech recognition.

  • Problem

    Structured-output tasks require modeling rich dependencies within outputs and their relationships to structured inputs, while fixed-dimensional encoder–decoder context vectors may discard input structure.

  • Method

    The paper reviews systems that use trained soft attention to select or weight different input subsets for successive outputs, alongside gated recurrent and convolutional neural networks.

  • Results

    Attention-based systems show strong performance across machine translation, image captioning, video description, and speech recognition, including over 50% relative improvement for English-to-French translation and 90% with a very large target vocabulary.

  • Takeaways & Limitations

    Attention provides a shared approach for relating structured inputs to structured outputs across several multimedia and sequence-generation tasks.

Abstract

from arXiv · show

Whereas deep neural networks were first mostly used for classification tasks, they are rapidly expanding in the realm of structured output problems, where the observed target is composed of multiple random variables that have a rich joint distribution, given the input. We focus in this paper on the case where the input also has a rich structure and the input and output structures are somehow related. We describe systems that learn to attend to different places in the input, for each element of the output, for a variety of tasks: machine translation, image caption generation, video clip description and speech recognition. All these systems are based on a shared set of building blocks: gated recurrent neural networks and convolutional neural networks, along with trained attention mechanisms. We report on experimental results with these systems, showing impressively good performance and the advantage of the attention mechanism.

I. INTRODUCTION

The paper addresses structured-output tasks whose input and output structures are related, focusing on soft attention for aligning input subsets with successive output elements. It reviews recurrent and convolutional building blocks used across translation, multimedia description, and speech tasks.

  • Structured-output tasks require modeling dependencies within the output sequence, not merely mapping each input to an isolated label.
  • Alignment relates sub-elements of structured inputs to corresponding sub-elements of structured outputs, such as words to semantic source parts or image regions.
  • Soft attention sequentially focuses on input subsets conditioned on the system state and previously attended subsets.
  • Unlike hard attention, soft attention weights all input subsets, sacrificing computational savings while enabling efficient gradient-backpropagation learning.
  • The reviewed systems use recurrent networks, including GRUs and LSTMs, whose gating units control information flow across time steps.

B. RNN-LM: Recurrent Neural Network Language Modeling

Recurrent neural language models represent sentence probabilities as sequential next-word predictions. Conditioning those predictions on an input context extends the model to translation, image captioning, and video description.

  • Language modeling learns the probability distribution over multi-word natural-language sentences.
  • Sentence probability decomposes into conditional next-word probabilities given all preceding words.
  • An RNN language model predicts the probability distribution of the next word at each time step, using its hidden state to summarize preceding symbols.
  • Exact sentence samples can be generated by iterative stochastic sampling, while beam search approximately finds high-probability sentences.
  • Conditional recurrent language models generate outputs such as captions, video descriptions, and translations from additional contexts.

C. Deep Convolutional Network

Deep convolutional networks extract spatial feature representations from images through convolution, nonlinear activation, pooling, and fully connected layers. Their learned intermediate representations can support tasks beyond image classification.

  • CNNs are feedforward networks designed for two-dimensional images, typically combining convolutional and fully connected layers.
  • Convolution applies local filters to image neighborhoods and produces nonlinear feature activations.
  • Local max-pooling reduces convolutional dimensionality and summarizes neighboring activations, producing local translation invariance.
  • After convolutional layers, the final feature map is flattened and passed through fully connected nonlinear layers to produce the output.
  • CNN-based entries have dominated the ImageNet classification track since 2012, and intermediate network representations can be reused for other tasks.

D. Transfer Learning with Deep Convolutional Network

The encoder–decoder framework maps structured inputs to structured outputs, using input-dependent encoders and output-appropriate decoders. Its simple fixed-dimensional context representation does not preserve input structure, motivating attention-based multimedia description.

  • Encoder–decoder networks handle mappings between highly structured inputs and outputs.
  • The encoder converts input x into a continuous-space context representation c, while the decoder generates output y conditioned on c.
  • Encoder choice depends on the input modality: CNNs suit images, whereas RNNs suit sentences.
  • For natural-language outputs, an RNN decoder models the description; other output types can use different decoders.
  • Applications include machine translation, image captioning, and video description using RNN or CNN encoders with language-model decoders.
  • Simple encoder–decoder models use fixed-dimensional, unstructured context vectors that need not preserve spatial, temporal, or spatio-temporal input structure.

1) Motivation:

Simple encoder–decoder models struggle with long or information-rich inputs because they compress everything into one fixed-dimensional vector. Attention addresses this bottleneck while exposing input–output alignment.

  • 13?
  • Performance of a simple neural machine translation encoder–decoder rapidly degraded as source-sentence length increased.
  • A fixed-dimensional context vector limits capacity and leaves the simple encoder–decoder difficult to interpret.
  • Attention lets the encoder return a structured context set of fixed-size vectors localized to spatial, temporal, or spatio-temporal input regions.
  • At each decoder step, an attention model scores context vectors against the decoder’s current hidden state and derives attention weights.
  • Weighted context computation frees the encoder from compressing variable-length inputs into one fixed-dimensional vector.
  • Visualizing attention weights provides an inspectable relation between input regions and output predictions.

C. Learning

The attention-based encoder–decoder is trained by maximizing the log-likelihood of the training set over all trainable model parameters.

  • Training maximizes the log-likelihood of the given training set with respect to the model parameters.
  • With weighted-sum attention, the model is differentiable and its log-likelihood gradient can be computed using backpropagation.
  • Stochastic gradient descent can iteratively update the trainable parameters to maximize log-likelihood.

1) Maximum Likelihood Learning:

Hard attention introduces discrete stochastic decisions that block direct backpropagation, so learning uses a lower-bound objective and sampled gradient estimates.

  • Hard attention’s discrete stochastic decisions have zero derivatives, preventing backpropagation from providing how to improve focus decisions.
  • With stochastic variables r, the log-likelihood is rewritten in terms of the stochastic computation and a lower bound is derived.
  • The gradient estimator is often approximated by Monte Carlo sampling.
  • High estimator variance motivates variance-reduction methods such as baselines and variance normalization.
  • After gradient estimation, standard gradient-based iterative optimization can approximately maximize the log-likelihood.

IV. APPLICATIONS

The applications section reviews attention-based encoder–decoder systems for structured-output tasks, detailing neural machine translation as a principal example. The translation model uses bidirectional encoding, attention over contextual states, and conditional recurrent decoding.

  • Attention-based encoder–decoder models are applied to multimedia description generation tasks, including machine translation and related structured-output problems.
  • The neural machine translation encoder uses a bidirectional recurrent network to produce forward and backward state sequences for the source sentence.
  • Forward and backward states are concatenated at each step to form a context set used by the attention-based decoder.
  • Bidirectional context is crucial for content-based attention because repeated source words otherwise cannot be distinguished using content alone.

2) Experimental Result:

The reviewed experiments cover attention-based translation and image caption generation. Results report strong translation improvements, competitive human-evaluation performance for captioning, and interpretable attention over spatial image regions.

  • Experimental Result:: More than 50% relative improvement was achieved on English-to-French translation with fixed model size, while a very large target vocabulary yielded 90% improvement over the no-attention baseline.
  • Experimental Result:: An attention-based translation model with a monolingual language model outperformed existing statistical systems on several reported language pairs and domains.
  • Image Caption Generation: Image captioning uses convolutional feature maps as multiple spatial context vectors, allowing attention to operate over image regions rather than one fully connected representation.
  • Experimental Result:: The attention-based captioning system was evaluated on Flickr 8K, Flickr 30K, and MS COCO, including automatic metrics and human evaluation.
  • Experimental Result:: The captioning approach ranked third on human-oriented measures, but eighth on CIDEr and ninth on BLEU in the Microsoft COCO challenge.
  • Experimental Result:: The attention-based image captioning model was reported to be highly interpretable compared with simple encoder–decoder models.

C. Video Description Generation

Video description models use attention to align generated words with temporal and motion-related video representations. Combining frame-wise and 3-D convolutional encoders captures complementary global and local temporal structure, and the resulting model outperforms a simple encoder–decoder model.

  • Model Description: Attention selects among per-frame feature vectors for each output symbol, allowing the model to capture global temporal structure across a video clip.The frame-wise encoder preserves all per-frame context vectors instead of averaging them.
  • Model Description: A 3-D convolutional encoder applies filters across spatial and temporal dimensions to model local motion statistics rather than appearance.Its activation vectors from the last convolutional layer serve as context, complementing global structure from the 2-D frame-wise encoder.
  • Model Description: The attention-based video description model combines frame-wise and 3-D convolutional encoders and uses a conditional RNN language-model decoder.The combined encoders were reported as beneficial on Youtube2Text and Montreal DVS.
  • Experimental Result: On Youtube2Text and Montreal DVS, the attention-based model outperforms the simple encoder–decoder model.The evaluation is summarized in Table IV.
  • Experimental Result: Attention weights provide a way to inspect which video content the model uses while generating each description.Figure 7 shows attention weights beneath video frames alongside generated and ground-truth descriptions.

2) Experimental Result:

The reviewed applications extend attention-based models beyond multimedia description to speech recognition and other structured-output tasks. Reported results include better generalization for location-aware speech attention and state-of-the-art matching for attention-based parsing.

  • Experimental Result: On TIMIT, location-aware attention can outperform conventional fully neural speech recognition and improves generalization error.Table V reports phoneme error rates for the evaluated systems.
  • Experimental Result: Speech attention weights can be visualized, showing attention to a roughly correct speech window each time the model generates a phoneme.Figure 8 displays these weights together with ground-truth phone locations.
  • Experimental Result: Attention-based parsing can match existing state-of-the-art parsers when generating linearized parse trees from natural-language sentences.The comparison concerns parsers that are often highly domain-specific.

2) Discrete Optimization–Pointer Network:

Attention-based encoder–decoder mechanisms are also applied to discrete optimization, question answering, and other structured tasks. These extensions use attention to select input symbols or relevant facts, while visualized alignments can emerge without explicit mapping supervision.

  • Discrete Optimization–Pointer Network: In discrete optimization, the decoder predicts which source symbol or node should be chosen at each time step instead of generating ordinary output symbols.For TSP, the selected cities form a route covering the input graph.
  • Discrete Optimization–Pointer Network: Pointer-style attention applies to discrete optimization problems whose solutions are expressed as subsets of the input symbols, including sorting.The paper identifies this generalization from the cited approach.
  • Discrete Optimization–Pointer Network: In question answering, attention over encoded fact vectors lets the model focus on facts relevant to the question.Each instance contains a set of facts and a question represented in continuous space.
  • Discrete Optimization–Pointer Network: Content-based attention computes relevance for spatial, temporal, or spatio-temporal input regions, whereas location-based attention directly returns an attended region.Location-based attention has been used for handwritten text and image object recognition.
  • Discrete Optimization–Pointer Network: The attention mechanism or its variants are proposed as applicable beyond multimedia input, including memory control and other AI-related tasks.The paper cites neural Turing machines, memory selection, and pointer networks as examples.
  • Discrete Optimization–Pointer Network: Across the reviewed applications, attention-based models infer alignments between different modalities without explicit supervision of the mapping.The paper reports that these alignments agree well with intuition in visualized examples.
Loading 1507.01053v1…